2024-04-15T22:08:53.894 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-15T22:08:53.894 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657551 --verbose --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} --owner scheduled_teuthology@teuthology --name teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi -- /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657551/orig.config.yaml 2024-04-15T22:08:53.923 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T22:08:54.000 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657551 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7657551' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 workunit: branch: squid sha1: 461a84e6fbbf089011eafd13b98169fb9e9b8cc9 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 7775 sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 sleep_before_teardown: 0 subset: 18/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 461a84e6fbbf089011eafd13b98169fb9e9b8cc9 targets: smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCPI1D4wbeFTYbyUH82+GuJSNRt54o9VFq15ChczDGJZ/bsLn4/BLY20WoTa6eLTd8Pxp4+1Pcfcri2pTKQz7CU= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGQnmaIcuCYUx6jXt9aYBab0fCtYtGH6H5NO5Z7BKyKIMsi8Tnm7Qaxg+kBz0lEDsAKEUJYnyZBpgAyU0bHMrVI= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-15_21:08:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-15T22:08:54.000 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa; will attempt to use it 2024-04-15T22:08:54.001 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks 2024-04-15T22:08:54.001 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-15T22:08:54.002 INFO:teuthology.task.internal:Checking packages... 2024-04-15T22:08:54.026 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a9a752df26c63acad72e1b3569fd79a515ca0765' 2024-04-15T22:08:54.026 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T22:08:54.026 INFO:teuthology.packaging:ref: None 2024-04-15T22:08:54.026 INFO:teuthology.packaging:tag: None 2024-04-15T22:08:54.026 INFO:teuthology.packaging:branch: squid 2024-04-15T22:08:54.026 INFO:teuthology.packaging:sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:08:54.027 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-15T22:08:54.330 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2114.ga9a752df 2024-04-15T22:08:54.331 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-15T22:08:54.338 INFO:teuthology.task.internal:no buildpackages task found 2024-04-15T22:08:54.338 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-15T22:08:54.345 INFO:teuthology.task.internal:Saving configuration 2024-04-15T22:08:54.357 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-15T22:08:54.363 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T22:08:54.388 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657551', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 22:01:20.270610', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCPI1D4wbeFTYbyUH82+GuJSNRt54o9VFq15ChczDGJZ/bsLn4/BLY20WoTa6eLTd8Pxp4+1Pcfcri2pTKQz7CU='} 2024-04-15T22:08:54.412 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657551', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 22:01:20.269350', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGQnmaIcuCYUx6jXt9aYBab0fCtYtGH6H5NO5Z7BKyKIMsi8Tnm7Qaxg+kBz0lEDsAKEUJYnyZBpgAyU0bHMrVI='} 2024-04-15T22:08:54.412 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-15T22:08:54.418 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-04-15T22:08:54.418 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-04-15T22:08:54.418 INFO:teuthology.run_tasks:Running task console_log... 2024-04-15T22:08:54.521 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f17846c60d0>, signals=[15]) 2024-04-15T22:08:54.522 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-15T22:08:54.527 INFO:teuthology.task.internal:Opening connections... 2024-04-15T22:08:54.528 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2024-04-15T22:08:54.529 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:08:54.617 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-04-15T22:08:54.617 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:08:54.703 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-15T22:08:54.710 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2024-04-15T22:08:54.727 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2024-04-15T22:08:54.728 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2024-04-15T22:08:54.784 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="CentOS Stream" 2024-04-15T22:08:54.784 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="9" 2024-04-15T22:08:54.784 INFO:teuthology.orchestra.run.smithi092.stdout:ID="centos" 2024-04-15T22:08:54.785 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="rhel fedora" 2024-04-15T22:08:54.785 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="9" 2024-04-15T22:08:54.785 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el9" 2024-04-15T22:08:54.785 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T22:08:54.785 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2024-04-15T22:08:54.785 INFO:teuthology.orchestra.run.smithi092.stdout:LOGO="fedora-logo-icon" 2024-04-15T22:08:54.785 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T22:08:54.785 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://centos.org/" 2024-04-15T22:08:54.785 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T22:08:54.785 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T22:08:54.785 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T22:08:54.786 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2024-04-15T22:08:54.811 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-04-15T22:08:54.828 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-04-15T22:08:54.828 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-04-15T22:08:54.885 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="CentOS Stream" 2024-04-15T22:08:54.885 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="9" 2024-04-15T22:08:54.885 INFO:teuthology.orchestra.run.smithi176.stdout:ID="centos" 2024-04-15T22:08:54.885 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE="rhel fedora" 2024-04-15T22:08:54.885 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="9" 2024-04-15T22:08:54.885 INFO:teuthology.orchestra.run.smithi176.stdout:PLATFORM_ID="platform:el9" 2024-04-15T22:08:54.885 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T22:08:54.885 INFO:teuthology.orchestra.run.smithi176.stdout:ANSI_COLOR="0;31" 2024-04-15T22:08:54.885 INFO:teuthology.orchestra.run.smithi176.stdout:LOGO="fedora-logo-icon" 2024-04-15T22:08:54.885 INFO:teuthology.orchestra.run.smithi176.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T22:08:54.886 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://centos.org/" 2024-04-15T22:08:54.886 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T22:08:54.886 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T22:08:54.886 INFO:teuthology.orchestra.run.smithi176.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T22:08:54.886 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-04-15T22:08:54.902 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-15T22:08:54.912 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-15T22:08:55.008 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-15T22:08:55.009 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2024-04-15T22:08:55.011 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-04-15T22:08:55.026 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-15T22:08:55.035 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-15T22:08:55.035 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2024-04-15T22:08:55.067 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-04-15T22:08:55.150 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-15T22:08:55.202 INFO:teuthology.run_tasks:Running task kernel... 2024-04-15T22:08:55.215 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-15T22:08:55.215 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-15T22:08:55.216 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T22:08:55.216 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T22:08:55.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:08:55.232 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-04-15T22:08:55.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:08:55.233 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-04-15T22:08:55.289 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-361.el9.x86_64 2024-04-15T22:08:55.289 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.14.0-361.el9.x86_64 2024-04-15T22:08:55.289 DEBUG:teuthology.orchestra.run.smithi176:> sudo yum install -y kernel 2024-04-15T22:08:55.290 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-361.el9.x86_64 2024-04-15T22:08:55.290 INFO:teuthology.task.kernel:Running kernel on smithi092: 5.14.0-361.el9.x86_64 2024-04-15T22:08:55.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum install -y kernel 2024-04-15T22:08:57.423 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-04-15T22:08:58.369 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 4.8 MB/s | 6.7 MB 00:01 2024-04-15T22:08:59.166 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-04-15T22:08:59.977 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-04-15T22:09:00.081 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-15T22:09:02.826 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.0 MB 00:02 2024-04-15T22:09:05.314 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 19 MB 00:04 2024-04-15T22:09:06.218 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 19 MB 00:02 2024-04-15T22:09:09.831 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.2 MB 00:01 2024-04-15T22:09:11.427 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.2 MB 00:01 2024-04-15T22:09:11.440 INFO:teuthology.orchestra.run.smithi176.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-15T22:09:12.902 INFO:teuthology.orchestra.run.smithi176.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-15T22:09:13.044 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-04-15T22:09:14.937 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-15T22:09:17.875 INFO:teuthology.orchestra.run.smithi176.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-15T22:09:20.028 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-15T22:09:20.231 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T22:09:20.231 INFO:teuthology.orchestra.run.smithi176.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T22:09:20.315 INFO:teuthology.orchestra.run.smithi176.stdout:Dependencies resolved. 2024-04-15T22:09:20.331 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-04-15T22:09:20.331 INFO:teuthology.orchestra.run.smithi176.stdout: Package Arch Version Repository Size 2024-04-15T22:09:20.331 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-04-15T22:09:20.331 INFO:teuthology.orchestra.run.smithi176.stdout:Installing: 2024-04-15T22:09:20.331 INFO:teuthology.orchestra.run.smithi176.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T22:09:20.332 INFO:teuthology.orchestra.run.smithi176.stdout:Installing dependencies: 2024-04-15T22:09:20.332 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T22:09:20.332 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T22:09:20.332 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T22:09:20.332 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-15T22:09:20.332 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction Summary 2024-04-15T22:09:20.332 INFO:teuthology.orchestra.run.smithi176.stdout:================================================================================ 2024-04-15T22:09:20.332 INFO:teuthology.orchestra.run.smithi176.stdout:Install 4 Packages 2024-04-15T22:09:20.333 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-15T22:09:20.333 INFO:teuthology.orchestra.run.smithi176.stdout:Total download size: 77 M 2024-04-15T22:09:20.334 INFO:teuthology.orchestra.run.smithi176.stdout:Installed size: 123 M 2024-04-15T22:09:20.334 INFO:teuthology.orchestra.run.smithi176.stdout:Downloading Packages: 2024-04-15T22:09:20.501 INFO:teuthology.orchestra.run.smithi176.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.6 MB/s | 277 kB 00:00 2024-04-15T22:09:21.243 INFO:teuthology.orchestra.run.smithi176.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-04-15T22:09:22.060 INFO:teuthology.orchestra.run.smithi176.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 19 MB/s | 34 MB 00:01 2024-04-15T22:09:22.310 INFO:teuthology.orchestra.run.smithi176.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 15 MB/s | 28 MB 00:01 2024-04-15T22:09:22.313 INFO:teuthology.orchestra.run.smithi176.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:09:22.313 INFO:teuthology.orchestra.run.smithi176.stdout:Total 39 MB/s | 77 MB 00:01 2024-04-15T22:09:22.314 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction check 2024-04-15T22:09:22.385 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T22:09:22.385 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T22:09:22.477 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-04-15T22:09:22.489 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-15T22:09:22.489 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-04-15T22:09:22.489 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-15T22:09:22.489 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-04-15T22:09:22.489 INFO:teuthology.orchestra.run.smithi092.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T22:09:22.489 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-04-15T22:09:22.489 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T22:09:22.489 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T22:09:22.489 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T22:09:22.489 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:09:22.489 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-04-15T22:09:22.490 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-04-15T22:09:22.490 INFO:teuthology.orchestra.run.smithi092.stdout:Install 4 Packages 2024-04-15T22:09:22.490 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:09:22.490 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 77 M 2024-04-15T22:09:22.490 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 123 M 2024-04-15T22:09:22.490 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-04-15T22:09:22.654 INFO:teuthology.orchestra.run.smithi092.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.7 MB/s | 277 kB 00:00 2024-04-15T22:09:22.684 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction check succeeded. 2024-04-15T22:09:22.684 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction test 2024-04-15T22:09:23.470 INFO:teuthology.orchestra.run.smithi092.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 15 MB/s | 15 MB 00:00 2024-04-15T22:09:23.509 INFO:teuthology.orchestra.run.smithi176.stdout:Transaction test succeeded. 2024-04-15T22:09:23.509 INFO:teuthology.orchestra.run.smithi176.stdout:Running transaction 2024-04-15T22:09:25.054 INFO:teuthology.orchestra.run.smithi092.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 13 MB/s | 34 MB 00:02 2024-04-15T22:09:25.320 INFO:teuthology.orchestra.run.smithi092.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 10 MB/s | 28 MB 00:02 2024-04-15T22:09:25.323 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-04-15T22:09:25.324 INFO:teuthology.orchestra.run.smithi092.stdout:Total 27 MB/s | 77 MB 00:02 2024-04-15T22:09:25.324 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-04-15T22:09:25.700 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-04-15T22:09:25.700 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-04-15T22:09:25.802 INFO:teuthology.orchestra.run.smithi176.stdout: Preparing : 1/1 2024-04-15T22:09:26.393 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T22:09:26.492 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:09:26.534 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-04-15T22:09:26.534 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-04-15T22:09:28.832 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-04-15T22:09:29.213 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:09:29.343 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:09:29.506 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T22:09:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:09:32.331 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:09:32.473 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:09:34.042 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:09:34.175 INFO:teuthology.orchestra.run.smithi176.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:09:37.197 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:09:37.230 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:09:38.868 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:09:41.973 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:08.200 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:08.200 INFO:teuthology.orchestra.run.smithi176.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T22:10:08.200 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-15T22:10:08.245 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:08.584 INFO:teuthology.orchestra.run.smithi176.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:08.584 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T22:10:08.585 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:10:08.585 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:10:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-15T22:10:09.001 INFO:teuthology.orchestra.run.smithi176.stdout:Installed: 2024-04-15T22:10:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T22:10:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T22:10:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T22:10:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T22:10:09.001 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-15T22:10:09.001 INFO:teuthology.orchestra.run.smithi176.stdout:Complete! 2024-04-15T22:10:09.321 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T22:10:09.475 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T22:10:09.476 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T22:10:09.476 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T22:10:09.476 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T22:10:09.476 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T22:10:09.476 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-15T22:10:09.476 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T22:10:09.476 DEBUG:teuthology.orchestra.run.smithi176:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T22:10:09.556 INFO:teuthology.orchestra.run.smithi176.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T22:10:09.556 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T22:10:09.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo rpm -qi grub2-tools 2024-04-15T22:10:09.593 INFO:teuthology.orchestra.run.smithi176.stdout:Name : grub2-tools 2024-04-15T22:10:09.593 INFO:teuthology.orchestra.run.smithi176.stdout:Epoch : 1 2024-04-15T22:10:09.593 INFO:teuthology.orchestra.run.smithi176.stdout:Version : 2.06 2024-04-15T22:10:09.593 INFO:teuthology.orchestra.run.smithi176.stdout:Release : 68.el9 2024-04-15T22:10:09.594 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: x86_64 2024-04-15T22:10:09.594 INFO:teuthology.orchestra.run.smithi176.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T22:10:09.594 INFO:teuthology.orchestra.run.smithi176.stdout:Group : Unspecified 2024-04-15T22:10:09.594 INFO:teuthology.orchestra.run.smithi176.stdout:Size : 8264254 2024-04-15T22:10:09.594 INFO:teuthology.orchestra.run.smithi176.stdout:License : GPLv3+ 2024-04-15T22:10:09.594 INFO:teuthology.orchestra.run.smithi176.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T22:10:09.594 INFO:teuthology.orchestra.run.smithi176.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T22:10:09.594 INFO:teuthology.orchestra.run.smithi176.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T22:10:09.594 INFO:teuthology.orchestra.run.smithi176.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T22:10:09.594 INFO:teuthology.orchestra.run.smithi176.stdout:Packager : builder@centos.org 2024-04-15T22:10:09.594 INFO:teuthology.orchestra.run.smithi176.stdout:Vendor : CentOS 2024-04-15T22:10:09.594 INFO:teuthology.orchestra.run.smithi176.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T22:10:09.595 INFO:teuthology.orchestra.run.smithi176.stdout:Summary : Support tools for GRUB. 2024-04-15T22:10:09.595 INFO:teuthology.orchestra.run.smithi176.stdout:Description : 2024-04-15T22:10:09.595 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-15T22:10:09.595 INFO:teuthology.orchestra.run.smithi176.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T22:10:09.595 INFO:teuthology.orchestra.run.smithi176.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T22:10:09.595 INFO:teuthology.orchestra.run.smithi176.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T22:10:09.595 INFO:teuthology.orchestra.run.smithi176.stdout:hardware devices. 2024-04-15T22:10:09.595 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-15T22:10:09.595 INFO:teuthology.orchestra.run.smithi176.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T22:10:09.596 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T22:10:09.596 INFO:teuthology.task.kernel:Updating grub on smithi176 to boot 5.14.0-438.el9.x86_64 2024-04-15T22:10:09.596 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T22:10:10.217 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2024-04-15T22:10:11.065 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:11.065 INFO:teuthology.orchestra.run.smithi092.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T22:10:11.065 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:10:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T22:10:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T22:10:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T22:10:11.669 INFO:teuthology.orchestra.run.smithi176.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T22:10:11.699 INFO:teuthology.orchestra.run.smithi176.stderr:done 2024-04-15T22:10:11.702 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-04-15T22:10:11.717 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.kn6UXemCJN 2024-04-15T22:10:11.718 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kn6UXemCJN 2024-04-15T22:10:11.786 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.kn6UXemCJN 2024-04-15T22:10:11.878 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T22:10:11.879 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:10:11.879 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-04-15T22:10:11.879 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T22:10:11.879 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T22:10:11.879 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T22:10:11.879 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T22:10:11.879 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:10:11.879 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-04-15T22:10:11.929 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.kn6UXemCJN is 6KB 2024-04-15T22:10:11.978 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.kn6UXemCJN 2024-04-15T22:10:11.992 DEBUG:teuthology.orchestra.run.smithi176:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T22:10:12.057 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T22:10:12.057 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T22:10:12.057 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T22:10:12.057 INFO:teuthology.orchestra.run.smithi176.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T22:10:12.058 DEBUG:teuthology.orchestra.run.smithi176:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T22:10:12.190 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2024-04-15T22:10:12.205 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T22:10:12.308 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T22:10:12.309 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T22:10:12.309 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T22:10:12.309 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T22:10:12.309 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T22:10:12.309 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-15T22:10:12.309 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T22:10:12.310 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T22:10:12.389 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T22:10:12.390 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T22:10:12.390 DEBUG:teuthology.orchestra.run.smithi092:> sudo rpm -qi grub2-tools 2024-04-15T22:10:12.434 INFO:teuthology.orchestra.run.smithi092.stdout:Name : grub2-tools 2024-04-15T22:10:12.434 INFO:teuthology.orchestra.run.smithi092.stdout:Epoch : 1 2024-04-15T22:10:12.434 INFO:teuthology.orchestra.run.smithi092.stdout:Version : 2.06 2024-04-15T22:10:12.434 INFO:teuthology.orchestra.run.smithi092.stdout:Release : 68.el9 2024-04-15T22:10:12.434 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: x86_64 2024-04-15T22:10:12.434 INFO:teuthology.orchestra.run.smithi092.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T22:10:12.435 INFO:teuthology.orchestra.run.smithi092.stdout:Group : Unspecified 2024-04-15T22:10:12.435 INFO:teuthology.orchestra.run.smithi092.stdout:Size : 8264254 2024-04-15T22:10:12.435 INFO:teuthology.orchestra.run.smithi092.stdout:License : GPLv3+ 2024-04-15T22:10:12.435 INFO:teuthology.orchestra.run.smithi092.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T22:10:12.435 INFO:teuthology.orchestra.run.smithi092.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T22:10:12.435 INFO:teuthology.orchestra.run.smithi092.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T22:10:12.435 INFO:teuthology.orchestra.run.smithi092.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T22:10:12.435 INFO:teuthology.orchestra.run.smithi092.stdout:Packager : builder@centos.org 2024-04-15T22:10:12.435 INFO:teuthology.orchestra.run.smithi092.stdout:Vendor : CentOS 2024-04-15T22:10:12.435 INFO:teuthology.orchestra.run.smithi092.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T22:10:12.435 INFO:teuthology.orchestra.run.smithi092.stdout:Summary : Support tools for GRUB. 2024-04-15T22:10:12.435 INFO:teuthology.orchestra.run.smithi092.stdout:Description : 2024-04-15T22:10:12.435 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:10:12.436 INFO:teuthology.orchestra.run.smithi092.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T22:10:12.436 INFO:teuthology.orchestra.run.smithi092.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T22:10:12.436 INFO:teuthology.orchestra.run.smithi092.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T22:10:12.436 INFO:teuthology.orchestra.run.smithi092.stdout:hardware devices. 2024-04-15T22:10:12.436 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:10:12.436 INFO:teuthology.orchestra.run.smithi092.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T22:10:12.437 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T22:10:12.437 INFO:teuthology.task.kernel:Updating grub on smithi092 to boot 5.14.0-438.el9.x86_64 2024-04-15T22:10:12.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T22:10:12.998 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2024-04-15T22:10:14.347 INFO:teuthology.orchestra.run.smithi092.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T22:10:14.376 INFO:teuthology.orchestra.run.smithi092.stderr:done 2024-04-15T22:10:14.378 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-04-15T22:10:14.393 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.8BJRs0698C 2024-04-15T22:10:14.393 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8BJRs0698C 2024-04-15T22:10:14.471 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.8BJRs0698C 2024-04-15T22:10:14.612 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.8BJRs0698C is 6KB 2024-04-15T22:10:14.661 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.8BJRs0698C 2024-04-15T22:10:14.675 DEBUG:teuthology.orchestra.run.smithi092:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T22:10:14.740 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T22:10:14.740 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T22:10:14.740 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T22:10:14.740 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T22:10:14.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T22:10:14.901 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2024-04-15T22:10:42.220 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-15T22:10:42.220 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-15T22:10:42.221 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:10:44.909 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-15T22:10:44.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-15T22:10:44.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:11:00.767 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-04-15T22:11:03.330 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-04-15T22:11:09.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-15T22:11:09.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:11:12.333 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-15T22:11:12.334 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:11:12.834 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-04-15T22:11:15.394 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-04-15T22:11:24.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-15T22:11:24.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:11:27.398 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-15T22:11:27.399 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:11:30.466 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-04-15T22:11:43.263 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-04-15T22:11:45.469 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-15T22:11:45.471 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:11:48.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-04-15T22:11:58.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-15T22:11:58.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:11:58.692 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-04-15T22:11:59.117 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-15T22:11:59.117 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-438.el9.x86_64"... 2024-04-15T22:11:59.118 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-04-15T22:11:59.132 INFO:teuthology.orchestra.run.smithi176.stdout:5.14.0-438.el9.x86_64 2024-04-15T22:11:59.133 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T22:11:59.133 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T22:11:59.133 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T22:12:00.134 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-15T22:12:00.134 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T22:12:00.194 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2024-04-15T22:12:00.230 DEBUG:teuthology.parallel:result is None 2024-04-15T22:12:06.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-15T22:12:06.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:12:06.886 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-04-15T22:12:07.263 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-15T22:12:07.263 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-15T22:12:07.264 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-04-15T22:12:07.280 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-438.el9.x86_64 2024-04-15T22:12:07.280 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T22:12:07.280 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T22:12:07.280 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T22:12:08.281 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-15T22:12:08.282 DEBUG:teuthology.orchestra.run.smithi092:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T22:12:08.361 INFO:teuthology.orchestra.run.smithi092.stdout:ttyS1 2024-04-15T22:12:08.389 DEBUG:teuthology.parallel:result is None 2024-04-15T22:12:08.389 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-15T22:12:08.402 INFO:teuthology.task.internal:Creating test directory... 2024-04-15T22:12:08.402 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T22:12:08.406 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T22:12:08.425 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-15T22:12:08.436 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-15T22:12:08.531 INFO:teuthology.task.internal:Creating archive directory... 2024-04-15T22:12:08.532 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T22:12:08.535 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T22:12:08.584 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-15T22:12:08.591 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-15T22:12:08.592 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T22:12:08.619 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T22:12:08.654 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T22:12:08.655 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T22:12:08.669 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T22:12:08.670 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T22:12:08.673 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-15T22:12:08.680 INFO:teuthology.task.internal:Configuring sudo... 2024-04-15T22:12:08.680 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T22:12:08.716 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T22:12:08.748 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-15T22:12:08.759 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-15T22:12:08.759 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T22:12:08.786 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T22:12:08.805 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T22:12:08.890 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T22:12:08.933 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T22:12:09.013 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T22:12:09.060 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-15T22:12:09.061 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T22:12:09.128 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T22:12:09.174 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T22:12:09.226 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T22:12:09.312 DEBUG:teuthology.orchestra.run.smithi176:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T22:12:09.360 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-15T22:12:09.360 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T22:12:09.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2024-04-15T22:12:09.434 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-04-15T22:12:09.486 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T22:12:09.540 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T22:12:10.015 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-15T22:12:10.023 INFO:teuthology.task.internal:Starting timer... 2024-04-15T22:12:10.023 INFO:teuthology.run_tasks:Running task pcp... 2024-04-15T22:12:10.034 INFO:teuthology.run_tasks:Running task selinux... 2024-04-15T22:12:10.043 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2024-04-15T22:12:10.092 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: 2024-04-15T22:12:10.094 DEBUG:teuthology.orchestra.run.smithi176:> sudo service auditd rotate 2024-04-15T22:12:10.171 INFO:teuthology.orchestra.run.smithi176.stdout:Rotating logs: 2024-04-15T22:12:10.173 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-15T22:12:10.173 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2024-04-15T22:12:10.210 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2024-04-15T22:12:10.210 DEBUG:teuthology.orchestra.run.smithi176:> /usr/sbin/getenforce 2024-04-15T22:12:10.235 INFO:teuthology.orchestra.run.smithi176.stdout:Permissive 2024-04-15T22:12:10.235 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi092.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi176.front.sepia.ceph.com': 'permissive'} 2024-04-15T22:12:10.235 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-15T22:12:10.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:12:10.279 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-15T22:12:10.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:12:10.310 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-15T22:12:10.310 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2024-04-15T22:12:10.354 DEBUG:teuthology.orchestra.run.smithi176:> sudo /usr/sbin/setenforce permissive 2024-04-15T22:12:10.387 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-15T22:12:10.397 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-15T22:12:10.415 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-15T22:12:10.417 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi092.front.sepia.ceph.com,smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-15T22:16:39.491 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi092.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2024-04-15T22:16:39.492 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-15T22:16:39.494 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:16:39.591 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-04-15T22:16:39.682 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-04-15T22:16:39.682 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-15T22:16:39.683 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T22:16:39.781 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-04-15T22:16:39.861 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-04-15T22:16:39.862 INFO:teuthology.run_tasks:Running task clock... 2024-04-15T22:16:39.872 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-15T22:16:39.872 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T22:16:39.873 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T22:16:39.875 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T22:16:39.875 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T22:16:39.905 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T22:16:39.922 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T22:16:39.934 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T22:16:39.952 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T22:16:39.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2024-04-15T22:16:39.968 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2024-04-15T22:16:39.983 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: ntpd: command not found 2024-04-15T22:16:39.985 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T22:16:39.998 INFO:teuthology.orchestra.run.smithi176.stdout:506 Cannot talk to daemon 2024-04-15T22:16:40.002 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T22:16:40.015 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T22:16:40.032 INFO:teuthology.orchestra.run.smithi176.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T22:16:40.056 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-04-15T22:16:40.059 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T22:16:40.059 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-04-15T22:16:40.059 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:16:40.059 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:16:40.059 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:16:40.059 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:16:40.084 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-04-15T22:16:40.088 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T22:16:40.088 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-04-15T22:16:40.088 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:16:40.088 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:16:40.088 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:16:40.088 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:16:40.089 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-15T22:16:40.197 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765'} 2024-04-15T22:16:40.198 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-15T22:16:40.198 INFO:tasks.cephadm:Cluster fsid is d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:16:40.198 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-15T22:16:40.199 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.92', 'mon.c': '[v2:172.21.15.92:3301,v1:172.21.15.92:6790]', 'mon.b': '172.21.15.176'} 2024-04-15T22:16:40.199 INFO:tasks.cephadm:First mon is mon.a on smithi092 2024-04-15T22:16:40.199 INFO:tasks.cephadm:First mgr is y 2024-04-15T22:16:40.199 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-15T22:16:40.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo hostname $(hostname -s) 2024-04-15T22:16:40.229 DEBUG:teuthology.orchestra.run.smithi176:> sudo hostname $(hostname -s) 2024-04-15T22:16:40.259 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-15T22:16:40.259 DEBUG:teuthology.orchestra.run.smithi092:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-15T22:16:40.504 INFO:teuthology.orchestra.run.smithi092.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 15 22:16 /home/ubuntu/cephtest/cephadm 2024-04-15T22:16:40.504 DEBUG:teuthology.orchestra.run.smithi176:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-15T22:16:40.880 INFO:teuthology.orchestra.run.smithi176.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 15 22:16 /home/ubuntu/cephtest/cephadm 2024-04-15T22:16:40.880 DEBUG:teuthology.orchestra.run.smithi092:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-15T22:16:40.897 DEBUG:teuthology.orchestra.run.smithi176:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-15T22:16:40.925 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-15T22:16:40.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-15T22:16:40.940 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-15T22:16:41.806 INFO:teuthology.orchestra.run.smithi176.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-15T22:16:41.827 INFO:teuthology.orchestra.run.smithi092.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-15T22:17:08.069 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-15T22:17:08.069 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-15T22:17:08.070 INFO:teuthology.orchestra.run.smithi092.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-15T22:17:08.070 INFO:teuthology.orchestra.run.smithi092.stdout: "repo_digests": [ 2024-04-15T22:17:08.070 INFO:teuthology.orchestra.run.smithi092.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-15T22:17:08.070 INFO:teuthology.orchestra.run.smithi092.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-15T22:17:08.070 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-15T22:17:08.070 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-15T22:17:11.803 INFO:teuthology.orchestra.run.smithi176.stdout:{ 2024-04-15T22:17:11.804 INFO:teuthology.orchestra.run.smithi176.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-15T22:17:11.804 INFO:teuthology.orchestra.run.smithi176.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-15T22:17:11.804 INFO:teuthology.orchestra.run.smithi176.stdout: "repo_digests": [ 2024-04-15T22:17:11.804 INFO:teuthology.orchestra.run.smithi176.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-15T22:17:11.804 INFO:teuthology.orchestra.run.smithi176.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-15T22:17:11.804 INFO:teuthology.orchestra.run.smithi176.stdout: ] 2024-04-15T22:17:11.804 INFO:teuthology.orchestra.run.smithi176.stdout:} 2024-04-15T22:17:11.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph 2024-04-15T22:17:11.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph 2024-04-15T22:17:11.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /etc/ceph 2024-04-15T22:17:11.928 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /etc/ceph 2024-04-15T22:17:11.958 INFO:tasks.cephadm:Writing seed config... 2024-04-15T22:17:11.959 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-04-15T22:17:11.959 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-15T22:17:11.960 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-15T22:17:11.960 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-04-15T22:17:11.960 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-15T22:17:11.960 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-15T22:17:11.960 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-15T22:17:11.960 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-15T22:17:11.960 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-15T22:17:11.961 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-15T22:17:11.961 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-15T22:17:11.985 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d537646e-fb75-11ee-bc8f-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-15T22:17:11.986 DEBUG:teuthology.orchestra.run.smithi092:mon.a> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.a.service 2024-04-15T22:17:12.029 DEBUG:teuthology.orchestra.run.smithi092:mgr.y> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mgr.y.service 2024-04-15T22:17:12.072 INFO:tasks.cephadm:Bootstrapping... 2024-04-15T22:17:12.072 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid d537646e-fb75-11ee-bc8f-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.92 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:17:12.269 INFO:teuthology.orchestra.run.smithi092.stderr:-------------------------------------------------------------------------------- 2024-04-15T22:17:12.269 INFO:teuthology.orchestra.run.smithi092.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'd537646e-fb75-11ee-bc8f-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.92', '--skip-admin-label'] 2024-04-15T22:17:12.293 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: 4.9.4-dev 2024-04-15T22:17:12.295 INFO:teuthology.orchestra.run.smithi092.stderr:Verifying podman|docker is present... 2024-04-15T22:17:12.318 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: 4.9.4-dev 2024-04-15T22:17:12.320 INFO:teuthology.orchestra.run.smithi092.stderr:Verifying lvm2 is present... 2024-04-15T22:17:12.320 INFO:teuthology.orchestra.run.smithi092.stderr:Verifying time synchronization is in place... 2024-04-15T22:17:12.327 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-15T22:17:12.333 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: inactive 2024-04-15T22:17:12.340 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: enabled 2024-04-15T22:17:12.346 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: active 2024-04-15T22:17:12.347 INFO:teuthology.orchestra.run.smithi092.stderr:Unit chronyd.service is enabled and running 2024-04-15T22:17:12.347 INFO:teuthology.orchestra.run.smithi092.stderr:Repeating the final host check... 2024-04-15T22:17:12.368 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: 4.9.4-dev 2024-04-15T22:17:12.370 INFO:teuthology.orchestra.run.smithi092.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-15T22:17:12.370 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl is present 2024-04-15T22:17:12.370 INFO:teuthology.orchestra.run.smithi092.stderr:lvcreate is present 2024-04-15T22:17:12.376 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-15T22:17:12.382 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: inactive 2024-04-15T22:17:12.389 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: enabled 2024-04-15T22:17:12.395 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: active 2024-04-15T22:17:12.396 INFO:teuthology.orchestra.run.smithi092.stderr:Unit chronyd.service is enabled and running 2024-04-15T22:17:12.396 INFO:teuthology.orchestra.run.smithi092.stderr:Host looks OK 2024-04-15T22:17:12.396 INFO:teuthology.orchestra.run.smithi092.stderr:Cluster fsid: d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:12.396 INFO:teuthology.orchestra.run.smithi092.stderr:Acquiring lock 140545897921888 on /run/cephadm/d537646e-fb75-11ee-bc8f-c7b262605968.lock 2024-04-15T22:17:12.396 INFO:teuthology.orchestra.run.smithi092.stderr:Lock 140545897921888 acquired on /run/cephadm/d537646e-fb75-11ee-bc8f-c7b262605968.lock 2024-04-15T22:17:12.397 INFO:teuthology.orchestra.run.smithi092.stderr:Verifying IP 172.21.15.92 port 3300 ... 2024-04-15T22:17:12.397 INFO:teuthology.orchestra.run.smithi092.stderr:Verifying IP 172.21.15.92 port 6789 ... 2024-04-15T22:17:12.397 INFO:teuthology.orchestra.run.smithi092.stderr:Base mon IP is 172.21.15.92, final addrv is [v2:172.21.15.92:3300,v1:172.21.15.92:6789] 2024-04-15T22:17:12.399 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.92 metric 100 2024-04-15T22:17:12.400 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.92 metric 100 2024-04-15T22:17:12.402 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-15T22:17:12.403 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-15T22:17:12.403 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1789sec hoplimit 64 pref medium 2024-04-15T22:17:12.405 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-15T22:17:12.405 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-15T22:17:12.405 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-15T22:17:12.406 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-15T22:17:12.406 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:cd45/64 scope link 2024-04-15T22:17:12.406 INFO:teuthology.orchestra.run.smithi092.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-15T22:17:12.406 INFO:teuthology.orchestra.run.smithi092.stderr:Mon IP `172.21.15.92` is in CIDR network `172.21.0.0/20` 2024-04-15T22:17:12.406 INFO:teuthology.orchestra.run.smithi092.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-15T22:17:12.407 INFO:teuthology.orchestra.run.smithi092.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-15T22:17:12.508 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-15T22:17:12.997 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Getting image source signatures 2024-04-15T22:17:13.015 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-15T22:17:13.015 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-15T22:17:13.015 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-15T22:17:13.015 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-15T22:17:13.016 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-15T22:17:13.016 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-15T22:17:13.163 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: Writing manifest to image destination 2024-04-15T22:17:13.316 INFO:teuthology.orchestra.run.smithi092.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-15T22:17:13.709 INFO:teuthology.orchestra.run.smithi092.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-15T22:17:14.239 INFO:teuthology.orchestra.run.smithi092.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-15T22:17:14.239 INFO:teuthology.orchestra.run.smithi092.stderr:Extracting ceph user uid/gid from container image... 2024-04-15T22:17:14.631 INFO:teuthology.orchestra.run.smithi092.stderr:stat: 167 167 2024-04-15T22:17:15.094 INFO:teuthology.orchestra.run.smithi092.stderr:Creating initial keys... 2024-04-15T22:17:15.561 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph-authtool: AQBrpx1m8f7OGxAAh6khkmUWnj3iMDvkRwBOwg== 2024-04-15T22:17:16.441 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph-authtool: AQBspx1mErUqFxAA4FEpbCkc2fPxuNtkWz9xFg== 2024-04-15T22:17:17.388 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph-authtool: AQBtpx1m142FERAA6u3CcEsDqJHyKg2KXu3iTQ== 2024-04-15T22:17:17.978 INFO:teuthology.orchestra.run.smithi092.stderr:Creating initial monmap... 2024-04-15T22:17:18.352 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-15T22:17:18.352 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-15T22:17:18.352 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:18.352 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-15T22:17:18.958 INFO:teuthology.orchestra.run.smithi092.stderr:monmaptool for a [v2:172.21.15.92:3300,v1:172.21.15.92:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-15T22:17:18.958 INFO:teuthology.orchestra.run.smithi092.stderr:setting min_mon_release = octopus 2024-04-15T22:17:18.958 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/monmaptool: set fsid to d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:18.958 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-15T22:17:18.958 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-15T22:17:18.958 INFO:teuthology.orchestra.run.smithi092.stderr:Creating mon... 2024-04-15T22:17:19.797 INFO:teuthology.orchestra.run.smithi092.stderr:create mon.a on 2024-04-15T22:17:19.998 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-15T22:17:20.252 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d537646e-fb75-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-d537646e-fb75-11ee-bc8f-c7b262605968.target. 2024-04-15T22:17:20.252 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-d537646e-fb75-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-d537646e-fb75-11ee-bc8f-c7b262605968.target. 2024-04-15T22:17:20.554 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Failed to reset failed state of unit ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.a.service: Unit ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.a.service not loaded. 2024-04-15T22:17:20.562 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Created symlink /etc/systemd/system/ceph-d537646e-fb75-11ee-bc8f-c7b262605968.target.wants/ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.a.service → /etc/systemd/system/ceph-d537646e-fb75-11ee-bc8f-c7b262605968@.service. 2024-04-15T22:17:21.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:21 smithi092 ceph-mon[24460]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-15T22:17:21.244 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: disabled 2024-04-15T22:17:21.251 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: inactive 2024-04-15T22:17:21.252 INFO:teuthology.orchestra.run.smithi092.stderr:firewalld.service is not enabled 2024-04-15T22:17:21.252 INFO:teuthology.orchestra.run.smithi092.stderr:Not possible to enable service . firewalld.service is not available 2024-04-15T22:17:21.252 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for mon to start... 2024-04-15T22:17:21.252 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for mon... 2024-04-15T22:17:21.487 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:21 smithi092 bash[24441]: b1a1654c0ee7c65215a62e9c0d21ee2d1fbfc231791d08e733e9762fd817dfab 2024-04-15T22:17:21.487 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:21 smithi092 systemd[1]: Started Ceph mon.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:17:21.748 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: cluster: 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: id: d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: services: 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.566602s) 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: data: 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: pgs: 2024-04-15T22:17:21.749 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-04-15T22:17:22.274 INFO:teuthology.orchestra.run.smithi092.stderr:mon is available 2024-04-15T22:17:22.274 INFO:teuthology.orchestra.run.smithi092.stderr:Assimilating anything we can from ceph.conf... 2024-04-15T22:17:22.397 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:22 smithi092 ceph-mon[24460]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-15T22:17:22.397 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:22 smithi092 ceph-mon[24460]: monmap e1: 1 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0]} 2024-04-15T22:17:22.397 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:22 smithi092 ceph-mon[24460]: fsmap 2024-04-15T22:17:22.397 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:22 smithi092 ceph-mon[24460]: osdmap e1: 0 total, 0 up, 0 in 2024-04-15T22:17:22.397 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:22 smithi092 ceph-mon[24460]: mgrmap e1: no daemons active 2024-04-15T22:17:22.397 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:22 smithi092 ceph-mon[24460]: from='client.? 172.21.15.92:0/4179988988' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-15T22:17:22.746 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-04-15T22:17:22.746 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: [global] 2024-04-15T22:17:22.746 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: fsid = d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:22.746 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.92:3300,v1:172.21.15.92:6789] 2024-04-15T22:17:22.746 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-15T22:17:22.746 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-15T22:17:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-15T22:17:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-15T22:17:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-04-15T22:17:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: [mgr] 2024-04-15T22:17:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = True 2024-04-15T22:17:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-15T22:17:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-04-15T22:17:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: [osd] 2024-04-15T22:17:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-15T22:17:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-15T22:17:23.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:23 smithi092 ceph-mon[24460]: from='client.? 172.21.15.92:0/1695545842' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-15T22:17:23.351 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:23 smithi092 ceph-mon[24460]: from='client.? 172.21.15.92:0/1695545842' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-15T22:17:23.354 INFO:teuthology.orchestra.run.smithi092.stderr:Generating new minimal ceph.conf... 2024-04-15T22:17:24.309 INFO:teuthology.orchestra.run.smithi092.stderr:Restarting the monitor... 2024-04-15T22:17:24.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:24 smithi092 ceph-mon[24460]: from='client.? 172.21.15.92:0/1862664798' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:17:24.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:24 smithi092 systemd[1]: Stopping Ceph mon.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:17:24.634 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:24 smithi092 bash[24676]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon.a" found: no such container 2024-04-15T22:17:24.634 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:24 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a[24456]: 2024-04-15T22:17:24.459+0000 7fd9f20be700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:17:24.634 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:24 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a[24456]: 2024-04-15T22:17:24.459+0000 7fd9f20be700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-15T22:17:24.942 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:24 smithi092 podman[24687]: 2024-04-15 22:17:24.633418684 +0000 UTC m=+0.266125988 container died b1a1654c0ee7c65215a62e9c0d21ee2d1fbfc231791d08e733e9762fd817dfab (image=quay.io/ceph/ceph:v17.2.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, distribution-scope=public, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, version=8, vcs-type=git, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-15T22:17:24.942 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:24 smithi092 podman[24687]: 2024-04-15 22:17:24.941963475 +0000 UTC m=+0.574670785 container cleanup b1a1654c0ee7c65215a62e9c0d21ee2d1fbfc231791d08e733e9762fd817dfab (image=quay.io/ceph/ceph:v17.2.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, distribution-scope=public, RELEASE=HEAD, ceph=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.openshift.tags=base centos centos-stream, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux ) 2024-04-15T22:17:25.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:24 smithi092 bash[24687]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a 2024-04-15T22:17:25.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:24 smithi092 bash[24714]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon.a" found: no such container 2024-04-15T22:17:26.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:25 smithi092 podman[24702]: 2024-04-15 22:17:25.870445292 +0000 UTC m=+1.233372381 container remove b1a1654c0ee7c65215a62e9c0d21ee2d1fbfc231791d08e733e9762fd817dfab (image=quay.io/ceph/ceph:v17.2.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a, io.openshift.expose-services=, RELEASE=HEAD, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, release=754, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-15T22:17:26.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:25 smithi092 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-15T22:17:26.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:25 smithi092 systemd[1]: Stopped Ceph mon.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:17:26.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:25 smithi092 systemd[1]: Starting Ceph mon.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:17:26.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:26 smithi092 podman[24783]: 2024-04-15 22:17:26.016654162 +0000 UTC m=+0.018732396 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-15T22:17:27.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:27 smithi092 podman[24783]: 2024-04-15 22:17:27.054738991 +0000 UTC m=+1.056817209 container create 5e439f1bb752bb2192198e1ffe51fa2382d8db60294210f0565d6c20771214da (image=quay.io/ceph/ceph:v17.2.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, release=754, io.openshift.tags=base centos centos-stream, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, name=centos-stream, distribution-scope=public, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-15T22:17:29.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 podman[24783]: 2024-04-15 22:17:29.032977392 +0000 UTC m=+3.035055613 container init 5e439f1bb752bb2192198e1ffe51fa2382d8db60294210f0565d6c20771214da (image=quay.io/ceph/ceph:v17.2.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, distribution-scope=public, GIT_BRANCH=HEAD, ceph=True, vcs-type=git, name=centos-stream, RELEASE=HEAD, GIT_CLEAN=True, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870) 2024-04-15T22:17:29.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 podman[24783]: 2024-04-15 22:17:29.036537643 +0000 UTC m=+3.038615864 container start 5e439f1bb752bb2192198e1ffe51fa2382d8db60294210f0565d6c20771214da (image=quay.io/ceph/ceph:v17.2.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a, io.openshift.expose-services=, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.buildah.version=1.19.8, release=754, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, version=8, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-15T22:17:29.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: set uid:gid to 167:167 (ceph:ceph) 2024-04-15T22:17:29.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-15T22:17:29.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: pidfile_write: ignore empty --pid-file 2024-04-15T22:17:29.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: load: jerasure load: lrc 2024-04-15T22:17:29.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: RocksDB version: 6.15.5 2024-04-15T22:17:29.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-15T22:17:29.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Compile date Apr 18 2022 2024-04-15T22:17:29.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: DB SUMMARY 2024-04-15T22:17:29.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: DB Session ID: CJ7FUTSC0OBTHXH7S19U 2024-04-15T22:17:29.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: CURRENT file: CURRENT 2024-04-15T22:17:29.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: IDENTITY file: IDENTITY 2024-04-15T22:17:29.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-15T22:17:29.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-15T22:17:29.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82637 ; 2024-04-15T22:17:29.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.error_if_exists: 0 2024-04-15T22:17:29.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.create_if_missing: 0 2024-04-15T22:17:29.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.paranoid_checks: 1 2024-04-15T22:17:29.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-15T22:17:29.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.env: 0x555e32ec6860 2024-04-15T22:17:29.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.fs: Posix File System 2024-04-15T22:17:29.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.info_log: 0x555e33bd1dc0 2024-04-15T22:17:29.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_file_opening_threads: 16 2024-04-15T22:17:29.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.statistics: (nil) 2024-04-15T22:17:29.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.use_fsync: 0 2024-04-15T22:17:29.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_log_file_size: 0 2024-04-15T22:17:29.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-15T22:17:29.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-15T22:17:29.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.keep_log_file_num: 1000 2024-04-15T22:17:29.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.recycle_log_file_num: 0 2024-04-15T22:17:29.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.allow_fallocate: 1 2024-04-15T22:17:29.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.allow_mmap_reads: 0 2024-04-15T22:17:29.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.allow_mmap_writes: 0 2024-04-15T22:17:29.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.use_direct_reads: 0 2024-04-15T22:17:29.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-15T22:17:29.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.create_missing_column_families: 0 2024-04-15T22:17:29.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.db_log_dir: 2024-04-15T22:17:29.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.advise_random_on_open: 1 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.db_write_buffer_size: 0 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.write_buffer_manager: 0x555e33cc22a0 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-15T22:17:29.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.rate_limiter: (nil) 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.wal_recovery_mode: 2 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.enable_thread_tracking: 0 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.enable_pipelined_write: 0 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.unordered_write: 0 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.row_cache: None 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.wal_filter: None 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.allow_ingest_behind: 0 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.preserve_deletes: 0 2024-04-15T22:17:29.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.two_write_queues: 0 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.manual_wal_flush: 0 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.atomic_flush: 0 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.log_readahead_size: 0 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.best_efforts_recovery: 0 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.allow_data_in_errors: 0 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.db_host_id: __hostname__ 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_background_jobs: 2 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_background_compactions: -1 2024-04-15T22:17:29.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_subcompactions: 1 2024-04-15T22:17:29.472 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-15T22:17:29.472 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-15T22:17:29.472 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-15T22:17:29.472 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_total_wal_size: 0 2024-04-15T22:17:29.473 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-15T22:17:29.473 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-15T22:17:29.473 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-15T22:17:29.473 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-15T22:17:29.473 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_open_files: -1 2024-04-15T22:17:29.473 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.bytes_per_sync: 0 2024-04-15T22:17:29.473 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-15T22:17:29.473 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-15T22:17:29.473 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compaction_readahead_size: 0 2024-04-15T22:17:29.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_background_flushes: -1 2024-04-15T22:17:29.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Compression algorithms supported: 2024-04-15T22:17:29.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-15T22:17:29.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: kZSTD supported: 0 2024-04-15T22:17:29.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: kXpressCompression supported: 0 2024-04-15T22:17:29.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: kLZ4HCCompression supported: 1 2024-04-15T22:17:29.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: kLZ4Compression supported: 1 2024-04-15T22:17:29.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: kBZip2Compression supported: 0 2024-04-15T22:17:29.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: kZlibCompression supported: 1 2024-04-15T22:17:29.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: kSnappyCompression supported: 1 2024-04-15T22:17:29.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-15T22:17:29.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-04-15T22:17:29.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.merge_operator: 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compaction_filter: None 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compaction_filter_factory: None 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.sst_partitioner_factory: None 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555e33b9dd00) 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout: cache_index_and_filter_blocks: 1 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout: pin_top_level_index_and_filter: 1 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout: index_type: 0 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout: data_block_index_type: 0 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout: index_shortening: 1 2024-04-15T22:17:29.475 INFO:journalctl@ceph.mon.a.smithi092.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-15T22:17:29.476 INFO:journalctl@ceph.mon.a.smithi092.stdout: hash_index_allow_collision: 1 2024-04-15T22:17:29.476 INFO:journalctl@ceph.mon.a.smithi092.stdout: checksum: 1 2024-04-15T22:17:29.476 INFO:journalctl@ceph.mon.a.smithi092.stdout: no_block_cache: 0 2024-04-15T22:17:29.476 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache: 0x555e33c08170 2024-04-15T22:17:29.476 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache_name: BinnedLRUCache 2024-04-15T22:17:29.476 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache_options: 2024-04-15T22:17:29.477 INFO:journalctl@ceph.mon.a.smithi092.stdout: capacity : 536870912 2024-04-15T22:17:29.477 INFO:journalctl@ceph.mon.a.smithi092.stdout: num_shard_bits : 4 2024-04-15T22:17:29.477 INFO:journalctl@ceph.mon.a.smithi092.stdout: strict_capacity_limit : 0 2024-04-15T22:17:29.477 INFO:journalctl@ceph.mon.a.smithi092.stdout: high_pri_pool_ratio: 0.000 2024-04-15T22:17:29.477 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache_compressed: (nil) 2024-04-15T22:17:29.477 INFO:journalctl@ceph.mon.a.smithi092.stdout: persistent_cache: (nil) 2024-04-15T22:17:29.477 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_size: 4096 2024-04-15T22:17:29.477 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_size_deviation: 10 2024-04-15T22:17:29.477 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_restart_interval: 16 2024-04-15T22:17:29.477 INFO:journalctl@ceph.mon.a.smithi092.stdout: index_block_restart_interval: 1 2024-04-15T22:17:29.477 INFO:journalctl@ceph.mon.a.smithi092.stdout: metadata_block_size: 4096 2024-04-15T22:17:29.477 INFO:journalctl@ceph.mon.a.smithi092.stdout: partition_filters: 0 2024-04-15T22:17:29.477 INFO:journalctl@ceph.mon.a.smithi092.stdout: use_delta_encoding: 1 2024-04-15T22:17:29.478 INFO:journalctl@ceph.mon.a.smithi092.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-15T22:17:29.478 INFO:journalctl@ceph.mon.a.smithi092.stdout: whole_key_filtering: 1 2024-04-15T22:17:29.478 INFO:journalctl@ceph.mon.a.smithi092.stdout: verify_compression: 0 2024-04-15T22:17:29.478 INFO:journalctl@ceph.mon.a.smithi092.stdout: read_amp_bytes_per_bit: 0 2024-04-15T22:17:29.478 INFO:journalctl@ceph.mon.a.smithi092.stdout: format_version: 4 2024-04-15T22:17:29.478 INFO:journalctl@ceph.mon.a.smithi092.stdout: enable_index_compression: 1 2024-04-15T22:17:29.478 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_align: 0 2024-04-15T22:17:29.478 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.write_buffer_size: 33554432 2024-04-15T22:17:29.478 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_write_buffer_number: 2 2024-04-15T22:17:29.478 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compression: NoCompression 2024-04-15T22:17:29.478 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.bottommost_compression: Disabled 2024-04-15T22:17:29.478 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.prefix_extractor: nullptr 2024-04-15T22:17:29.478 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.num_levels: 7 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compression_opts.level: 32767 2024-04-15T22:17:29.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compression_opts.strategy: 0 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compression_opts.enabled: false 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.target_file_size_base: 67108864 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-15T22:17:29.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-15T22:17:29.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-15T22:17:29.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-15T22:17:29.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-15T22:17:29.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-15T22:17:29.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-15T22:17:29.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-15T22:17:29.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.arena_block_size: 4194304 2024-04-15T22:17:29.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-15T22:17:29.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-15T22:17:29.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-15T22:17:29.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.disable_auto_compactions: 0 2024-04-15T22:17:29.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-15T22:17:29.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-15T22:17:29.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-15T22:17:29.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-15T22:17:29.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-15T22:17:29.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-15T22:17:29.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-15T22:17:29.482 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.table_properties_collectors: 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.inplace_update_support: 0 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.bloom_locality: 0 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.max_successive_merges: 0 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.paranoid_file_checks: 0 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.force_consistency_checks: 1 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.report_bg_io_stats: 0 2024-04-15T22:17:29.483 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.ttl: 2592000 2024-04-15T22:17:29.484 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-15T22:17:29.484 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.enable_blob_files: false 2024-04-15T22:17:29.484 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.min_blob_size: 0 2024-04-15T22:17:29.484 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.blob_file_size: 268435456 2024-04-15T22:17:29.484 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-15T22:17:29.484 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-15T22:17:29.484 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-15T22:17:29.484 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-15T22:17:29.484 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-15T22:17:29.485 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-15T22:17:29.485 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713219449062383, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-15T22:17:29.485 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-15T22:17:29.485 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-15T22:17:29.485 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713219449063077, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79187, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77450, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71759, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713219449, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "411330d1-81a0-4779-a095-3981e3c1a6eb", "db_session_id": "CJ7FUTSC0OBTHXH7S19U"}} 2024-04-15T22:17:29.485 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-15T22:17:29.485 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713219449063399, "job": 1, "event": "recovery_finished"} 2024-04-15T22:17:29.485 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-15T22:17:29.485 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x555e33beea80 2024-04-15T22:17:29.485 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: DB pointer 0x555e33bfe000 2024-04-15T22:17:29.485 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-15T22:17:29.486 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-15T22:17:29.486 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** DB Stats ** 2024-04-15T22:17:29.486 INFO:journalctl@ceph.mon.a.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T22:17:29.486 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-15T22:17:29.486 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T22:17:29.486 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T22:17:29.486 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-15T22:17:29.486 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-15T22:17:29.486 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T22:17:29.486 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-04-15T22:17:29.486 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** Compaction Stats [default] ** 2024-04-15T22:17:29.486 INFO:journalctl@ceph.mon.a.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T22:17:29.486 INFO:journalctl@ceph.mon.a.smithi092.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.0 0.00 0.00 1 0.000 0 0 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.0 0.00 0.00 1 0.000 0 0 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.0 0.00 0.00 1 0.000 0 0 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** Compaction Stats [default] ** 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 160.0 0.00 0.00 1 0.000 0 0 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T22:17:29.487 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative compaction: 0.00 GB write, 25.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval compaction: 0.00 GB write, 25.43 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** Compaction Stats [default] ** 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: L0 2/0 79.09 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.0 0.00 0.00 1 0.000 0 0 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: Sum 2/0 79.09 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 160.0 0.00 0.00 1 0.000 0 0 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** Compaction Stats [default] ** 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:17:29.488 INFO:journalctl@ceph.mon.a.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 160.0 0.00 0.00 1 0.000 0 0 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative compaction: 0.00 GB write, 25.14 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: starting mon.a rank 0 at public addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] at bind addrs [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: mon.a@-1(???) e1 preinit fsid d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: mon.a@-1(???).mds e1 new map 2024-04-15T22:17:29.489 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: mon.a@-1(???).mds e1 print_map 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout: e1 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout: legacy client fscid: -1 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout: No filesystems configured 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: expand_channel_meta expand map: {default=false} 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: expand_channel_meta from 'false' to 'false' 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: expand_channel_meta expanded map: {default=false} 2024-04-15T22:17:29.490 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: expand_channel_meta expand map: {default=info} 2024-04-15T22:17:29.491 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: expand_channel_meta from 'info' to 'info' 2024-04-15T22:17:29.491 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: expand_channel_meta expanded map: {default=info} 2024-04-15T22:17:29.491 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: expand_channel_meta expand map: {default=daemon} 2024-04-15T22:17:29.491 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-15T22:17:29.491 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: expand_channel_meta expanded map: {default=daemon} 2024-04-15T22:17:29.491 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: expand_channel_meta expand map: {default=debug} 2024-04-15T22:17:29.491 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: expand_channel_meta from 'debug' to 'debug' 2024-04-15T22:17:29.491 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: expand_channel_meta expanded map: {default=debug} 2024-04-15T22:17:29.491 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-15T22:17:29.491 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: monmap e1: 1 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0]} 2024-04-15T22:17:29.491 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: fsmap 2024-04-15T22:17:29.491 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: osdmap e1: 0 total, 0 up, 0 in 2024-04-15T22:17:29.491 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:29 smithi092 ceph-mon[24813]: mgrmap e1: no daemons active 2024-04-15T22:17:30.039 INFO:teuthology.orchestra.run.smithi092.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-15T22:17:30.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:30 smithi092 bash[24783]: 5e439f1bb752bb2192198e1ffe51fa2382d8db60294210f0565d6c20771214da 2024-04-15T22:17:30.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:30 smithi092 systemd[1]: Started Ceph mon.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:17:34.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:34 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/4182085510' entity='client.admin' 2024-04-15T22:17:38.241 INFO:teuthology.orchestra.run.smithi092.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-15T22:17:38.241 INFO:teuthology.orchestra.run.smithi092.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:17:38.241 INFO:teuthology.orchestra.run.smithi092.stderr:Creating mgr... 2024-04-15T22:17:38.241 INFO:teuthology.orchestra.run.smithi092.stderr:Verifying port 9283 ... 2024-04-15T22:17:38.420 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Failed to reset failed state of unit ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mgr.y.service: Unit ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mgr.y.service not loaded. 2024-04-15T22:17:38.427 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: Created symlink /etc/systemd/system/ceph-d537646e-fb75-11ee-bc8f-c7b262605968.target.wants/ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-d537646e-fb75-11ee-bc8f-c7b262605968@.service. 2024-04-15T22:17:39.095 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: disabled 2024-04-15T22:17:39.104 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: inactive 2024-04-15T22:17:39.104 INFO:teuthology.orchestra.run.smithi092.stderr:firewalld.service is not enabled 2024-04-15T22:17:39.104 INFO:teuthology.orchestra.run.smithi092.stderr:Not possible to enable service . firewalld.service is not available 2024-04-15T22:17:39.111 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: disabled 2024-04-15T22:17:39.118 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: inactive 2024-04-15T22:17:39.119 INFO:teuthology.orchestra.run.smithi092.stderr:firewalld.service is not enabled 2024-04-15T22:17:39.119 INFO:teuthology.orchestra.run.smithi092.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-15T22:17:39.119 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for mgr to start... 2024-04-15T22:17:39.120 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for mgr... 2024-04-15T22:17:39.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:39 smithi092 systemd[1]: Started Ceph mgr.y for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:17:39.502 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:39 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:39.492+0000 7ff6e467b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T22:17:39.683 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-04-15T22:17:39.683 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-04-15T22:17:39.683 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "fsid": "d537646e-fb75-11ee-bc8f-c7b262605968", 2024-04-15T22:17:39.683 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "health": { 2024-04-15T22:17:39.683 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "checks": {}, 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mutes": [] 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum": [ 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 0 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "a" 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "monmap": { 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-15T22:17:39.684 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-15T22:17:39.685 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:39.685 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osdmap": { 2024-04-15T22:17:39.685 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:17:39.685 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-15T22:17:39.685 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-15T22:17:39.685 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-15T22:17:39.685 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-15T22:17:39.689 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-15T22:17:39.689 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-15T22:17:39.689 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:39.689 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "pgmap": { 2024-04-15T22:17:39.689 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-15T22:17:39.689 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-15T22:17:39.689 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-15T22:17:39.689 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-15T22:17:39.689 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-15T22:17:39.689 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-15T22:17:39.689 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-15T22:17:39.689 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-15T22:17:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "fsmap": { 2024-04-15T22:17:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:17:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-15T22:17:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-15T22:17:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-15T22:17:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "available": false, 2024-04-15T22:17:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-15T22:17:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "modules": [ 2024-04-15T22:17:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "iostat", 2024-04-15T22:17:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "nfs", 2024-04-15T22:17:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "restful" 2024-04-15T22:17:39.691 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-04-15T22:17:39.691 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:17:39.691 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:39.691 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "servicemap": { 2024-04-15T22:17:39.691 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:17:39.691 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "modified": "2024-04-15T22:17:21.181200+0000", 2024-04-15T22:17:39.691 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:17:39.691 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:39.691 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-15T22:17:39.691 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-04-15T22:17:39.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:39 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:39.603+0000 7ff6e467b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T22:17:40.255 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:40 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/1415903395' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T22:17:40.256 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:40.206+0000 7ff6e467b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T22:17:40.258 INFO:teuthology.orchestra.run.smithi092.stderr:mgr not available, waiting (1/15)... 2024-04-15T22:17:40.527 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:40.296+0000 7ff6e467b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T22:17:40.528 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:40.379+0000 7ff6e467b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T22:17:40.798 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:40.526+0000 7ff6e467b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T22:17:40.799 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:40.607+0000 7ff6e467b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T22:17:40.799 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:40.715+0000 7ff6e467b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T22:17:40.799 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:40.797+0000 7ff6e467b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T22:17:41.214 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:40.958+0000 7ff6e467b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T22:17:41.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:41 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:41.042+0000 7ff6e467b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T22:17:42.387 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:42 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:42.071+0000 7ff6e467b000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T22:17:42.387 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:42 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:42.199+0000 7ff6e467b000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T22:17:42.714 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:42 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:42.397+0000 7ff6e467b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T22:17:42.715 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:42 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:42.486+0000 7ff6e467b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T22:17:42.715 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:42 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:42.624+0000 7ff6e467b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T22:17:42.806 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-04-15T22:17:42.806 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-04-15T22:17:42.806 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "fsid": "d537646e-fb75-11ee-bc8f-c7b262605968", 2024-04-15T22:17:42.806 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "health": { 2024-04-15T22:17:42.806 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-15T22:17:42.806 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "checks": {}, 2024-04-15T22:17:42.806 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mutes": [] 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum": [ 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 0 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "a" 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum_age": 13, 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "monmap": { 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-15T22:17:42.807 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:42.808 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osdmap": { 2024-04-15T22:17:42.808 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:17:42.808 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-15T22:17:42.808 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-15T22:17:42.808 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-15T22:17:42.808 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-15T22:17:42.811 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-15T22:17:42.811 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-15T22:17:42.811 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "pgmap": { 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "fsmap": { 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-15T22:17:42.812 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "available": false, 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "modules": [ 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "iostat", 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "nfs", 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "restful" 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "servicemap": { 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "modified": "2024-04-15T22:17:21.181200+0000", 2024-04-15T22:17:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:17:42.814 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:42.814 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-15T22:17:42.814 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-04-15T22:17:43.136 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:42 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/676336682' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T22:17:43.136 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:42 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:42.954+0000 7ff6e467b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T22:17:43.432 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:43.177+0000 7ff6e467b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T22:17:43.432 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:43.261+0000 7ff6e467b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T22:17:43.433 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:43.431+0000 7ff6e467b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T22:17:43.447 INFO:teuthology.orchestra.run.smithi092.stderr:mgr not available, waiting (2/15)... 2024-04-15T22:17:44.630 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:44 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:44.340+0000 7ff6e467b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T22:17:44.631 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:44 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:44.629+0000 7ff6e467b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T22:17:44.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:44 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:44.722+0000 7ff6e467b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T22:17:44.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:44 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:44.871+0000 7ff6e467b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T22:17:45.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:45 smithi092 ceph-mon[24813]: Activating manager daemon y 2024-04-15T22:17:45.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:45 smithi092 ceph-mon[24813]: mgrmap e2: y(active, starting, since 0.00302747s) 2024-04-15T22:17:45.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:45 smithi092 ceph-mon[24813]: from='mgr.14100 172.21.15.92:0/1722722870' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:17:45.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:45 smithi092 ceph-mon[24813]: from='mgr.14100 172.21.15.92:0/1722722870' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:17:45.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:45 smithi092 ceph-mon[24813]: from='mgr.14100 172.21.15.92:0/1722722870' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:17:45.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:45 smithi092 ceph-mon[24813]: from='mgr.14100 172.21.15.92:0/1722722870' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:17:45.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:45 smithi092 ceph-mon[24813]: from='mgr.14100 172.21.15.92:0/1722722870' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T22:17:45.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:45 smithi092 ceph-mon[24813]: Manager daemon y is now available 2024-04-15T22:17:45.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:45 smithi092 ceph-mon[24813]: from='mgr.14100 172.21.15.92:0/1722722870' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:17:45.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:45 smithi092 ceph-mon[24813]: from='mgr.14100 172.21.15.92:0/1722722870' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:17:45.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:45 smithi092 ceph-mon[24813]: from='mgr.14100 172.21.15.92:0/1722722870' entity='mgr.y' 2024-04-15T22:17:45.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:45 smithi092 ceph-mon[24813]: from='mgr.14100 172.21.15.92:0/1722722870' entity='mgr.y' 2024-04-15T22:17:45.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:45 smithi092 ceph-mon[24813]: from='mgr.14100 172.21.15.92:0/1722722870' entity='mgr.y' 2024-04-15T22:17:45.947 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-04-15T22:17:45.947 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-04-15T22:17:45.948 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "fsid": "d537646e-fb75-11ee-bc8f-c7b262605968", 2024-04-15T22:17:45.948 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "health": { 2024-04-15T22:17:45.948 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-15T22:17:45.948 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "checks": {}, 2024-04-15T22:17:45.948 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mutes": [] 2024-04-15T22:17:45.948 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:45.948 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-15T22:17:45.948 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum": [ 2024-04-15T22:17:45.948 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 0 2024-04-15T22:17:45.948 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-04-15T22:17:45.948 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-15T22:17:45.948 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "a" 2024-04-15T22:17:45.949 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-04-15T22:17:45.949 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "quorum_age": 16, 2024-04-15T22:17:45.949 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "monmap": { 2024-04-15T22:17:45.949 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:17:45.949 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-15T22:17:45.949 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-15T22:17:45.949 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:45.949 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osdmap": { 2024-04-15T22:17:45.949 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:17:45.949 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-15T22:17:45.949 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-15T22:17:45.950 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-15T22:17:45.950 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-15T22:17:45.954 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-15T22:17:45.954 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-15T22:17:45.954 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:45.954 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "pgmap": { 2024-04-15T22:17:45.954 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-15T22:17:45.954 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-15T22:17:45.955 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-15T22:17:45.955 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-15T22:17:45.955 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-15T22:17:45.955 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-15T22:17:45.955 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-15T22:17:45.955 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-15T22:17:45.955 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:45.955 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "fsmap": { 2024-04-15T22:17:45.955 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:17:45.955 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-15T22:17:45.955 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-15T22:17:45.955 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:45.956 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-15T22:17:45.956 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "available": true, 2024-04-15T22:17:45.956 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-15T22:17:45.956 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "modules": [ 2024-04-15T22:17:45.956 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "iostat", 2024-04-15T22:17:45.956 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "nfs", 2024-04-15T22:17:45.956 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "restful" 2024-04-15T22:17:45.956 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ], 2024-04-15T22:17:45.956 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:17:45.956 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:45.956 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "servicemap": { 2024-04-15T22:17:45.956 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T22:17:45.957 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "modified": "2024-04-15T22:17:21.181200+0000", 2024-04-15T22:17:45.957 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "services": {} 2024-04-15T22:17:45.957 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: }, 2024-04-15T22:17:45.957 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-15T22:17:45.957 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-04-15T22:17:46.502 INFO:teuthology.orchestra.run.smithi092.stderr:mgr is available 2024-04-15T22:17:47.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:46 smithi092 ceph-mon[24813]: mgrmap e3: y(active, since 1.00561s) 2024-04-15T22:17:47.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:46 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/1032862635' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T22:17:47.040 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-04-15T22:17:47.040 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: [global] 2024-04-15T22:17:47.040 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: fsid = d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:17:47.040 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-15T22:17:47.040 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-15T22:17:47.040 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-15T22:17:47.040 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-15T22:17:47.041 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-04-15T22:17:47.041 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: [mgr] 2024-04-15T22:17:47.041 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-15T22:17:47.041 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 2024-04-15T22:17:47.041 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: [osd] 2024-04-15T22:17:47.041 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-15T22:17:47.041 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-15T22:17:47.608 INFO:teuthology.orchestra.run.smithi092.stderr:Enabling cephadm module... 2024-04-15T22:17:48.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:47 smithi092 ceph-mon[24813]: mgrmap e4: y(active, since 2s) 2024-04-15T22:17:48.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:47 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2776524106' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-15T22:17:48.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:47 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2776524106' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-15T22:17:49.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:48 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2242552329' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-15T22:17:49.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:48 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ignoring --setuser ceph since I am not root 2024-04-15T22:17:49.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:48 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ignoring --setgroup ceph since I am not root 2024-04-15T22:17:49.489 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:49 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:49.391+0000 7fee519e1000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T22:17:49.489 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:49 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:49.482+0000 7fee519e1000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T22:17:50.087 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:49 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2242552329' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-15T22:17:50.087 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:49 smithi092 ceph-mon[24813]: mgrmap e5: y(active, since 4s) 2024-04-15T22:17:50.189 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-04-15T22:17:50.189 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-15T22:17:50.189 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "available": true, 2024-04-15T22:17:50.189 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-15T22:17:50.189 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-15T22:17:50.190 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-04-15T22:17:50.413 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:50.086+0000 7fee519e1000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T22:17:50.413 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:50.176+0000 7fee519e1000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T22:17:50.413 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:50.264+0000 7fee519e1000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T22:17:50.413 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:50.412+0000 7fee519e1000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T22:17:50.685 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:50.492+0000 7fee519e1000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T22:17:50.685 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:50.600+0000 7fee519e1000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T22:17:50.685 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:50.685+0000 7fee519e1000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T22:17:50.803 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for the mgr to restart... 2024-04-15T22:17:50.803 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for mgr epoch 5... 2024-04-15T22:17:50.943 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:50 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/1451418690' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-15T22:17:50.944 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:50.855+0000 7fee519e1000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T22:17:50.944 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:50.942+0000 7fee519e1000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T22:17:52.302 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:51 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:51.987+0000 7fee519e1000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T22:17:52.302 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:52 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:52.114+0000 7fee519e1000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T22:17:52.302 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:52 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:52.301+0000 7fee519e1000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T22:17:52.715 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:52 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:52.386+0000 7fee519e1000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T22:17:52.715 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:52 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:52.519+0000 7fee519e1000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T22:17:53.132 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:52 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:52.833+0000 7fee519e1000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T22:17:53.132 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:53.048+0000 7fee519e1000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T22:17:53.132 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:53.131+0000 7fee519e1000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T22:17:53.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:53.298+0000 7fee519e1000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T22:17:54.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:54 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:54.202+0000 7fee519e1000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T22:17:54.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:54 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:54.494+0000 7fee519e1000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T22:17:54.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:54 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:54.587+0000 7fee519e1000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T22:17:54.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:54 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:17:54.736+0000 7fee519e1000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T22:17:55.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:55 smithi092 ceph-mon[24813]: Active manager daemon y restarted 2024-04-15T22:17:55.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:55 smithi092 ceph-mon[24813]: Activating manager daemon y 2024-04-15T22:17:55.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:55 smithi092 ceph-mon[24813]: osdmap e2: 0 total, 0 up, 0 in 2024-04-15T22:17:56.085 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-04-15T22:17:56.085 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-15T22:17:56.086 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "initialized": true 2024-04-15T22:17:56.086 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-04-15T22:17:56.389 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:55 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: [15/Apr/2024:22:17:55] ENGINE Bus STARTING 2024-04-15T22:17:56.389 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: [15/Apr/2024:22:17:56] ENGINE Serving on https://172.21.15.92:7150 2024-04-15T22:17:56.390 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: [15/Apr/2024:22:17:56] ENGINE Bus STARTED 2024-04-15T22:17:56.390 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: mgrmap e6: y(active, starting, since 0.337792s) 2024-04-15T22:17:56.390 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:17:56.390 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T22:17:56.390 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:17:56.390 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:17:56.390 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:17:56.390 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: Manager daemon y is now available 2024-04-15T22:17:56.390 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:17:56.391 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:17:56.391 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:17:56.391 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:17:56.391 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:17:56.391 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:17:56.391 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:17:56.391 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:56 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:17:56.609 INFO:teuthology.orchestra.run.smithi092.stderr:mgr epoch 5 is available 2024-04-15T22:17:56.609 INFO:teuthology.orchestra.run.smithi092.stderr:Setting orchestrator backend to cephadm... 2024-04-15T22:17:57.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:57 smithi092 ceph-mon[24813]: [15/Apr/2024:22:17:55] ENGINE Bus STARTING 2024-04-15T22:17:57.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:57 smithi092 ceph-mon[24813]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-15T22:17:57.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:57 smithi092 ceph-mon[24813]: mgrmap e7: y(active, since 1.34025s) 2024-04-15T22:17:57.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:57 smithi092 ceph-mon[24813]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-15T22:17:57.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:57 smithi092 ceph-mon[24813]: [15/Apr/2024:22:17:56] ENGINE Serving on https://172.21.15.92:7150 2024-04-15T22:17:57.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:57 smithi092 ceph-mon[24813]: [15/Apr/2024:22:17:56] ENGINE Bus STARTED 2024-04-15T22:17:57.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:57 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:17:57.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:57 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:17:58.431 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:58 smithi092 ceph-mon[24813]: mgrmap e8: y(active, since 2s) 2024-04-15T22:17:58.431 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:58 smithi092 ceph-mon[24813]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:17:58.431 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:58 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:17:58.431 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:58 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:17:58.432 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: value unchanged 2024-04-15T22:17:59.212 INFO:teuthology.orchestra.run.smithi092.stderr:Generating ssh key... 2024-04-15T22:17:59.455 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:17:59 smithi092 ceph-mon[24813]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:00.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-mon[24813]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:00.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-mon[24813]: Generating ssh key... 2024-04-15T22:18:00.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:18:00.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:18:00.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: Generating public/private rsa key pair. 2024-04-15T22:18:00.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: Your identification has been saved in /tmp/tmpnlee09r0/key. 2024-04-15T22:18:00.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: Your public key has been saved in /tmp/tmpnlee09r0/key.pub. 2024-04-15T22:18:00.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: The key fingerprint is: 2024-04-15T22:18:00.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: SHA256:lto9AQvRMz4GdjsMYHvvcNt6B+YgDIhrjUw2yc4THUw ceph-d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:18:00.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: The key's randomart image is: 2024-04-15T22:18:00.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: +---[RSA 3072]----+ 2024-04-15T22:18:00.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: | oEo... | 2024-04-15T22:18:00.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: | + .+.= | 2024-04-15T22:18:00.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: |..o.o.o*.+ | 2024-04-15T22:18:00.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: |.B.... oB+ | 2024-04-15T22:18:00.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: |*.= o..So. | 2024-04-15T22:18:00.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: |.B . oB.+o. | 2024-04-15T22:18:00.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: |. . ..+++. | 2024-04-15T22:18:00.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: | .o.. | 2024-04-15T22:18:00.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: | .. . | 2024-04-15T22:18:00.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: +----[SHA256]-----+ 2024-04-15T22:18:01.168 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8VdR7KRYiGOuUqBofEN3fi9VXoC/ReRP7SDcnLSfCfsbpsJguG8oBAwBVNEWzFuTsigUMuOPwfY2IqgP/ca0Tf29mC8FvN4y3X9tsmj6bWoNraaiwfwKDQxFkj3FdTDphla/4EMV3f1LcwejUrVS0+6B7OQhC/4RpQyACug7yiLnZqkhZX7dIZPvmR/cRU3GZMn2gjQ/8Tl+xkhxsg7UuY29NcL4y+GPUqsZRoavUSrnlsq4txVhkLvI5P+UrhOaTNF8ReQVQmXLxs0+eC/lvve7hSj14m+1C0cDxObkBPt7aW7DGL5Xi/S5TDTu224E5yTjjtfFiEvCdV09pGg1SlkYvxOymRgFNbvDluzEODuPzLf+1H+Wj1ERhsNjXPSS5jBmsKC+T25MajYnss6H72dozAqyBRntGvztWoES1WRQgnUtZY2pT7bngvqWnd3AeGk+duH4IbiYomXB6NW2laJsvCzmz+4Z+/fN8Ej7XblVBYKgoKr9nrBLBJQCoS8U= ceph-d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:18:01.790 INFO:teuthology.orchestra.run.smithi092.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-15T22:18:01.791 INFO:teuthology.orchestra.run.smithi092.stderr:Adding key to root@localhost authorized_keys... 2024-04-15T22:18:01.791 INFO:teuthology.orchestra.run.smithi092.stderr:Adding host smithi092... 2024-04-15T22:18:02.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:01 smithi092 ceph-mon[24813]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:03.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:02 smithi092 ceph-mon[24813]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi092", "addr": "172.21.15.92", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:03.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:02 smithi092 ceph-mon[24813]: Deploying cephadm binary to smithi092 2024-04-15T22:18:03.363 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: Added host 'smithi092' with addr '172.21.15.92' 2024-04-15T22:18:03.980 INFO:teuthology.orchestra.run.smithi092.stderr:Deploying unmanaged mon service... 2024-04-15T22:18:04.582 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:04 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:18:04.582 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:04 smithi092 ceph-mon[24813]: Added host smithi092 2024-04-15T22:18:04.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:04 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:04.583 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-15T22:18:05.313 INFO:teuthology.orchestra.run.smithi092.stderr:Deploying unmanaged mgr service... 2024-04-15T22:18:05.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:05 smithi092 ceph-mon[24813]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:05.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:05 smithi092 ceph-mon[24813]: Saving service mon spec with placement count:5 2024-04-15T22:18:05.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:05 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:18:06.079 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-15T22:18:07.255 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:07 smithi092 ceph-mon[24813]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:07.256 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:07 smithi092 ceph-mon[24813]: Saving service mgr spec with placement count:2 2024-04-15T22:18:07.256 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:07 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:18:07.256 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:07 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:18:08.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:08 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:18:08.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:08 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/1741896610' entity='client.admin' 2024-04-15T22:18:10.473 INFO:teuthology.orchestra.run.smithi092.stderr:Enabling the dashboard module... 2024-04-15T22:18:10.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:10 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:18:10.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:10 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:18:10.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:10 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:18:10.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:10 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' 2024-04-15T22:18:10.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:10 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi092", "caps": []}]: dispatch 2024-04-15T22:18:10.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:10 smithi092 ceph-mon[24813]: from='mgr.14118 172.21.15.92:0/3096088578' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi092", "caps": []}]': finished 2024-04-15T22:18:10.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:10 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2098094460' entity='client.admin' 2024-04-15T22:18:12.144 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:11 smithi092 ceph-mon[24813]: Deploying daemon agent.smithi092 on smithi092 2024-04-15T22:18:12.144 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:11 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/965632584' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-15T22:18:12.144 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ignoring --setuser ceph since I am not root 2024-04-15T22:18:12.144 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ignoring --setgroup ceph since I am not root 2024-04-15T22:18:12.444 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:12 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:12.220+0000 7f67bcbf7000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T22:18:12.444 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:12 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:12.310+0000 7f67bcbf7000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T22:18:13.018 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:12 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/965632584' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-15T22:18:13.018 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:12 smithi092 ceph-mon[24813]: mgrmap e9: y(active, since 16s) 2024-04-15T22:18:13.018 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:12 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:12.926+0000 7f67bcbf7000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T22:18:13.018 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:13.016+0000 7f67bcbf7000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T22:18:13.102 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-04-15T22:18:13.103 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-15T22:18:13.103 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "available": true, 2024-04-15T22:18:13.103 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-15T22:18:13.103 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-15T22:18:13.103 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-04-15T22:18:13.337 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:13.102+0000 7f67bcbf7000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T22:18:13.337 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:13.255+0000 7f67bcbf7000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T22:18:13.337 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:13.336+0000 7f67bcbf7000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T22:18:13.695 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:13.446+0000 7f67bcbf7000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T22:18:13.695 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:13.529+0000 7f67bcbf7000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T22:18:13.695 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:13.693+0000 7f67bcbf7000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T22:18:13.744 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for the mgr to restart... 2024-04-15T22:18:13.744 INFO:teuthology.orchestra.run.smithi092.stderr:Waiting for mgr epoch 9... 2024-04-15T22:18:13.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:13 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/1716346806' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-15T22:18:13.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:13.785+0000 7f67bcbf7000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T22:18:15.133 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:14 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:14.811+0000 7f67bcbf7000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T22:18:15.133 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:14 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:14.942+0000 7f67bcbf7000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T22:18:15.133 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:15 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:15.132+0000 7f67bcbf7000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T22:18:15.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:15 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:15.219+0000 7f67bcbf7000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T22:18:15.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:15 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:15.354+0000 7f67bcbf7000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T22:18:15.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:15 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:15.670+0000 7f67bcbf7000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T22:18:15.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:15 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:15.892+0000 7f67bcbf7000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T22:18:16.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:15 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:15.977+0000 7f67bcbf7000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T22:18:16.465 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:16 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:16.145+0000 7f67bcbf7000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T22:18:17.347 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:17 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:17.055+0000 7f67bcbf7000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T22:18:17.348 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:17 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:17.346+0000 7f67bcbf7000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T22:18:17.650 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:17 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:17.440+0000 7f67bcbf7000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T22:18:17.650 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:17 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:18:17.589+0000 7f67bcbf7000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T22:18:17.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:17 smithi092 ceph-mon[24813]: Active manager daemon y restarted 2024-04-15T22:18:17.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:17 smithi092 ceph-mon[24813]: Activating manager daemon y 2024-04-15T22:18:17.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:17 smithi092 ceph-mon[24813]: osdmap e3: 0 total, 0 up, 0 in 2024-04-15T22:18:17.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:17 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: [15/Apr/2024:22:18:17] ENGINE Bus STARTING 2024-04-15T22:18:18.364 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: [15/Apr/2024:22:18:18] ENGINE Serving on https://172.21.15.92:7150 2024-04-15T22:18:18.364 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: [15/Apr/2024:22:18:18] ENGINE Bus STARTED 2024-04-15T22:18:18.664 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: { 2024-04-15T22:18:18.664 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-15T22:18:18.664 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: "initialized": true 2024-04-15T22:18:18.664 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: } 2024-04-15T22:18:18.937 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-mon[24813]: mgrmap e10: y(active, starting, since 0.0557256s) 2024-04-15T22:18:18.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:18:18.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T22:18:18.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:18:18.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:18:18.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:18:18.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-mon[24813]: Manager daemon y is now available 2024-04-15T22:18:18.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:18.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:18.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:18.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:18:18.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:18:18.938 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:19.303 INFO:teuthology.orchestra.run.smithi092.stderr:mgr epoch 9 is available 2024-04-15T22:18:19.303 INFO:teuthology.orchestra.run.smithi092.stderr:Generating a dashboard self-signed certificate... 2024-04-15T22:18:19.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:19 smithi092 ceph-mon[24813]: [15/Apr/2024:22:18:17] ENGINE Bus STARTING 2024-04-15T22:18:19.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:19 smithi092 ceph-mon[24813]: [15/Apr/2024:22:18:18] ENGINE Serving on https://172.21.15.92:7150 2024-04-15T22:18:19.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:19 smithi092 ceph-mon[24813]: [15/Apr/2024:22:18:18] ENGINE Bus STARTED 2024-04-15T22:18:19.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:19 smithi092 ceph-mon[24813]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-15T22:18:19.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:19 smithi092 ceph-mon[24813]: mgrmap e11: y(active, since 1.0664s) 2024-04-15T22:18:19.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:19 smithi092 ceph-mon[24813]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-15T22:18:19.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:19 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:19.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:19 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:18:19.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:19 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:19.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:19 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi092", "caps": []}]: dispatch 2024-04-15T22:18:20.019 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-15T22:18:20.659 INFO:teuthology.orchestra.run.smithi092.stderr:Creating initial admin user... 2024-04-15T22:18:20.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:20 smithi092 ceph-mon[24813]: Deploying daemon agent.smithi092 on smithi092 2024-04-15T22:18:20.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:20 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:20.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:20 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:20.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:20 smithi092 ceph-mon[24813]: mgrmap e12: y(active, since 2s) 2024-04-15T22:18:21.498 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$NC0MsE0QAN6gMjksNUC1wekqa6woq6E1tbHvs4ytRzESvD2eEmPlq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713219501, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-15T22:18:21.812 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:21 smithi092 ceph-mon[24813]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:21.813 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:21 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:22.149 INFO:teuthology.orchestra.run.smithi092.stderr:Fetching dashboard port number... 2024-04-15T22:18:22.782 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: 8443 2024-04-15T22:18:22.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:22 smithi092 ceph-mon[24813]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:23.323 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: disabled 2024-04-15T22:18:23.330 INFO:teuthology.orchestra.run.smithi092.stderr:systemctl: inactive 2024-04-15T22:18:23.331 INFO:teuthology.orchestra.run.smithi092.stderr:firewalld.service is not enabled 2024-04-15T22:18:23.331 INFO:teuthology.orchestra.run.smithi092.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-15T22:18:23.333 INFO:teuthology.orchestra.run.smithi092.stderr:Ceph Dashboard is now available at: 2024-04-15T22:18:23.333 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-15T22:18:23.333 INFO:teuthology.orchestra.run.smithi092.stderr: URL: https://smithi092.front.sepia.ceph.com:8443/ 2024-04-15T22:18:23.333 INFO:teuthology.orchestra.run.smithi092.stderr: User: admin 2024-04-15T22:18:23.333 INFO:teuthology.orchestra.run.smithi092.stderr: Password: ua70hvcyec 2024-04-15T22:18:23.333 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-15T22:18:23.334 INFO:teuthology.orchestra.run.smithi092.stderr:Enabling autotune for osd_memory_target 2024-04-15T22:18:23.706 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:23 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/161574120' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-15T22:18:24.760 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:24 smithi092 ceph-mon[24813]: mgrmap e13: y(active, since 6s) 2024-04-15T22:18:25.397 INFO:teuthology.orchestra.run.smithi092.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-15T22:18:26.028 INFO:teuthology.orchestra.run.smithi092.stderr:You can access the Ceph CLI with: 2024-04-15T22:18:26.028 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-15T22:18:26.028 INFO:teuthology.orchestra.run.smithi092.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:18:26.028 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-15T22:18:26.028 INFO:teuthology.orchestra.run.smithi092.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-15T22:18:26.028 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-15T22:18:26.028 INFO:teuthology.orchestra.run.smithi092.stderr: ceph telemetry on 2024-04-15T22:18:26.028 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-15T22:18:26.028 INFO:teuthology.orchestra.run.smithi092.stderr:For more information see: 2024-04-15T22:18:26.028 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-15T22:18:26.028 INFO:teuthology.orchestra.run.smithi092.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-15T22:18:26.029 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-15T22:18:26.029 INFO:teuthology.orchestra.run.smithi092.stderr:Bootstrap complete. 2024-04-15T22:18:26.205 INFO:tasks.cephadm:Fetching config... 2024-04-15T22:18:26.205 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-15T22:18:26.205 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-15T22:18:26.221 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-15T22:18:26.221 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-15T22:18:26.221 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-15T22:18:26.277 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-15T22:18:26.277 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-15T22:18:26.277 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-15T22:18:26.392 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-15T22:18:26.393 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-15T22:18:26.393 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-15T22:18:26.449 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-15T22:18:26.450 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8VdR7KRYiGOuUqBofEN3fi9VXoC/ReRP7SDcnLSfCfsbpsJguG8oBAwBVNEWzFuTsigUMuOPwfY2IqgP/ca0Tf29mC8FvN4y3X9tsmj6bWoNraaiwfwKDQxFkj3FdTDphla/4EMV3f1LcwejUrVS0+6B7OQhC/4RpQyACug7yiLnZqkhZX7dIZPvmR/cRU3GZMn2gjQ/8Tl+xkhxsg7UuY29NcL4y+GPUqsZRoavUSrnlsq4txVhkLvI5P+UrhOaTNF8ReQVQmXLxs0+eC/lvve7hSj14m+1C0cDxObkBPt7aW7DGL5Xi/S5TDTu224E5yTjjtfFiEvCdV09pGg1SlkYvxOymRgFNbvDluzEODuPzLf+1H+Wj1ERhsNjXPSS5jBmsKC+T25MajYnss6H72dozAqyBRntGvztWoES1WRQgnUtZY2pT7bngvqWnd3AeGk+duH4IbiYomXB6NW2laJsvCzmz+4Z+/fN8Ej7XblVBYKgoKr9nrBLBJQCoS8U= ceph-d537646e-fb75-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-15T22:18:26.568 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:26 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/1579315135' entity='client.admin' 2024-04-15T22:18:26.679 INFO:teuthology.orchestra.run.smithi092.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8VdR7KRYiGOuUqBofEN3fi9VXoC/ReRP7SDcnLSfCfsbpsJguG8oBAwBVNEWzFuTsigUMuOPwfY2IqgP/ca0Tf29mC8FvN4y3X9tsmj6bWoNraaiwfwKDQxFkj3FdTDphla/4EMV3f1LcwejUrVS0+6B7OQhC/4RpQyACug7yiLnZqkhZX7dIZPvmR/cRU3GZMn2gjQ/8Tl+xkhxsg7UuY29NcL4y+GPUqsZRoavUSrnlsq4txVhkLvI5P+UrhOaTNF8ReQVQmXLxs0+eC/lvve7hSj14m+1C0cDxObkBPt7aW7DGL5Xi/S5TDTu224E5yTjjtfFiEvCdV09pGg1SlkYvxOymRgFNbvDluzEODuPzLf+1H+Wj1ERhsNjXPSS5jBmsKC+T25MajYnss6H72dozAqyBRntGvztWoES1WRQgnUtZY2pT7bngvqWnd3AeGk+duH4IbiYomXB6NW2laJsvCzmz+4Z+/fN8Ej7XblVBYKgoKr9nrBLBJQCoS8U= ceph-d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:18:26.838 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8VdR7KRYiGOuUqBofEN3fi9VXoC/ReRP7SDcnLSfCfsbpsJguG8oBAwBVNEWzFuTsigUMuOPwfY2IqgP/ca0Tf29mC8FvN4y3X9tsmj6bWoNraaiwfwKDQxFkj3FdTDphla/4EMV3f1LcwejUrVS0+6B7OQhC/4RpQyACug7yiLnZqkhZX7dIZPvmR/cRU3GZMn2gjQ/8Tl+xkhxsg7UuY29NcL4y+GPUqsZRoavUSrnlsq4txVhkLvI5P+UrhOaTNF8ReQVQmXLxs0+eC/lvve7hSj14m+1C0cDxObkBPt7aW7DGL5Xi/S5TDTu224E5yTjjtfFiEvCdV09pGg1SlkYvxOymRgFNbvDluzEODuPzLf+1H+Wj1ERhsNjXPSS5jBmsKC+T25MajYnss6H72dozAqyBRntGvztWoES1WRQgnUtZY2pT7bngvqWnd3AeGk+duH4IbiYomXB6NW2laJsvCzmz+4Z+/fN8Ej7XblVBYKgoKr9nrBLBJQCoS8U= ceph-d537646e-fb75-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-15T22:18:26.886 INFO:teuthology.orchestra.run.smithi176.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC8VdR7KRYiGOuUqBofEN3fi9VXoC/ReRP7SDcnLSfCfsbpsJguG8oBAwBVNEWzFuTsigUMuOPwfY2IqgP/ca0Tf29mC8FvN4y3X9tsmj6bWoNraaiwfwKDQxFkj3FdTDphla/4EMV3f1LcwejUrVS0+6B7OQhC/4RpQyACug7yiLnZqkhZX7dIZPvmR/cRU3GZMn2gjQ/8Tl+xkhxsg7UuY29NcL4y+GPUqsZRoavUSrnlsq4txVhkLvI5P+UrhOaTNF8ReQVQmXLxs0+eC/lvve7hSj14m+1C0cDxObkBPt7aW7DGL5Xi/S5TDTu224E5yTjjtfFiEvCdV09pGg1SlkYvxOymRgFNbvDluzEODuPzLf+1H+Wj1ERhsNjXPSS5jBmsKC+T25MajYnss6H72dozAqyBRntGvztWoES1WRQgnUtZY2pT7bngvqWnd3AeGk+duH4IbiYomXB6NW2laJsvCzmz+4Z+/fN8Ej7XblVBYKgoKr9nrBLBJQCoS8U= ceph-d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:18:26.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-15T22:18:27.354 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:27 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:18:27] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:18:27.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:27.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:27.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:27.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:27.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:27.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:28.952 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:30.904 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:30 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:18:30] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:18:31.774 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-15T22:18:31.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-15T22:18:31.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:31 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:31.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:31 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:31.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:31 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/4036714300' entity='client.admin' 2024-04-15T22:18:31.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:31 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:31.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:31 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:32.388 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:32 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:18:32] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:18:33.392 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:33 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:33.392 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:33 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:34.605 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi176 2024-04-15T22:18:34.605 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-15T22:18:34.605 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.conf 2024-04-15T22:18:34.624 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-15T22:18:34.624 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:18:34.681 INFO:tasks.cephadm:Adding host smithi176 to orchestrator... 2024-04-15T22:18:34.682 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch host add smithi176 2024-04-15T22:18:35.177 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:34 smithi092 ceph-mon[24813]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:35.177 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:34 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:35.177 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:34 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:35.177 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:34 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:18:35.177 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:34 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:18:35.177 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:34 smithi092 ceph-mon[24813]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-15T22:18:35.177 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:34 smithi092 ceph-mon[24813]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:18:35.177 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:34 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:35.177 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:34 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:37.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:36 smithi092 ceph-mon[24813]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi176", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:37.460 INFO:teuthology.orchestra.run.smithi092.stdout:Added host 'smithi176' with addr '172.21.15.176' 2024-04-15T22:18:38.070 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch host ls --format=json 2024-04-15T22:18:38.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: Deploying cephadm binary to smithi176 2024-04-15T22:18:38.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:38.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: Added host smithi176 2024-04-15T22:18:38.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:38.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:18:38.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:18:38.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:38.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:38.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:38.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:38.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:18:38.681 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:38.681 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi176", "caps": []}]: dispatch 2024-04-15T22:18:38.681 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi176", "caps": []}]': finished 2024-04-15T22:18:39.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:39 smithi092 ceph-mon[24813]: Deploying daemon agent.smithi176 on smithi176 2024-04-15T22:18:39.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:39.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:39.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:18:39.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:18:39.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:39.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:39.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:39.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:18:39.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:18:39.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:39.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:39 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:18:39] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:18:39.995 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:18:39.995 INFO:teuthology.orchestra.run.smithi092.stdout:[{"addr": "172.21.15.92", "hostname": "smithi092", "labels": [], "status": ""}, {"addr": "172.21.15.176", "hostname": "smithi176", "labels": [], "status": ""}] 2024-04-15T22:18:40.560 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:40 smithi092 ceph-mon[24813]: Updating smithi176:/etc/ceph/ceph.conf 2024-04-15T22:18:40.561 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:40 smithi092 ceph-mon[24813]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:18:40.561 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:40 smithi092 ceph-mon[24813]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:40.561 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:40 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:40.562 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-15T22:18:40.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd crush tunables default 2024-04-15T22:18:40.870 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:18:40] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:18:41.762 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:41 smithi092 ceph-mon[24813]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:18:41.762 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:41 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:41.762 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:41 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:42.013 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:41 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:18:41] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:18:42.801 INFO:teuthology.orchestra.run.smithi092.stderr:adjusted tunables profile to default 2024-04-15T22:18:42.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:42 smithi092 ceph-mon[24813]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:42.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:42 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:42.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:42 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:42.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:42 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/172400840' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-15T22:18:43.415 INFO:tasks.cephadm:Adding mon.a on smithi092 2024-04-15T22:18:43.415 INFO:tasks.cephadm:Adding mon.c on smithi092 2024-04-15T22:18:43.415 INFO:tasks.cephadm:Adding mon.b on smithi176 2024-04-15T22:18:43.415 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch apply mon '3;smithi092:172.21.15.92=a;smithi092:[v2:172.21.15.92:3301,v1:172.21.15.92:6790]=c;smithi176:172.21.15.176=b' 2024-04-15T22:18:43.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:43 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:43.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:43 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/172400840' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-15T22:18:43.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:43 smithi092 ceph-mon[24813]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T22:18:44.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:44 smithi092 ceph-mon[24813]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:45.284 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled mon update... 2024-04-15T22:18:45.816 DEBUG:teuthology.orchestra.run.smithi092:mon.c> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.c.service 2024-04-15T22:18:45.818 DEBUG:teuthology.orchestra.run.smithi176:mon.b> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.b.service 2024-04-15T22:18:45.825 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-15T22:18:45.825 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:18:46.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:46 smithi092 ceph-mon[24813]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi092:172.21.15.92=a;smithi092:[v2:172.21.15.92:3301,v1:172.21.15.92:6790]=c;smithi176:172.21.15.176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:46.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:46 smithi092 ceph-mon[24813]: Saving service mon spec with placement smithi092:172.21.15.92=a;smithi092:[v2:172.21.15.92:3301,v1:172.21.15.92:6790]=c;smithi176:172.21.15.176=b;count:3 2024-04-15T22:18:46.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:46 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:46.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:46 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:46.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:46 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:18:46.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:46 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:18:46.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:46 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:46.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:46 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:18:46.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:46 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:18:46.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:46 smithi092 ceph-mon[24813]: Deploying daemon mon.c on smithi092 2024-04-15T22:18:46.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:46 smithi092 ceph-mon[24813]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:47.552 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-15T22:18:47.552 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":1,"fsid":"d537646e-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:17:18.281141Z","created":"2024-04-15T22:17:18.281141Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3300","nonce":0},{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T22:18:47.554 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 1 2024-04-15T22:18:47.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:47 smithi092 ceph-mon[24813]: from='client.? 172.21.15.176:0/1279738944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:18:48.784 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[24813]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:48.785 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 podman[29677]: 2024-04-15 22:18:48.697375675 +0000 UTC m=+0.119899562 container create 69aa2c1b085d0d7f19b354ec43764b665296c7a116e1f949ce606ee1531567b1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-c, distribution-scope=public, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, release=754, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, vendor=Red Hat, Inc., ceph=True, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, io.openshift.expose-services=, name=centos-stream, maintainer=Guillaume Abrioux ) 2024-04-15T22:18:49.129 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-15T22:18:49.129 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:18:49.194 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 podman[29677]: 2024-04-15 22:18:48.891492616 +0000 UTC m=+0.314016504 container init 69aa2c1b085d0d7f19b354ec43764b665296c7a116e1f949ce606ee1531567b1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-c, vcs-type=git, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, ceph=True, io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, version=8, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-15T22:18:49.194 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 podman[29677]: 2024-04-15 22:18:48.894906682 +0000 UTC m=+0.317430569 container start 69aa2c1b085d0d7f19b354ec43764b665296c7a116e1f949ce606ee1531567b1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-c, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-15T22:18:49.194 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: set uid:gid to 167:167 (ceph:ceph) 2024-04-15T22:18:49.194 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-15T22:18:49.194 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: pidfile_write: ignore empty --pid-file 2024-04-15T22:18:49.194 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: load: jerasure load: lrc 2024-04-15T22:18:49.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: RocksDB version: 6.15.5 2024-04-15T22:18:49.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-15T22:18:49.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Compile date Apr 18 2022 2024-04-15T22:18:49.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: DB SUMMARY 2024-04-15T22:18:49.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: DB Session ID: FGJPYK049DHEN9Y0L0D8 2024-04-15T22:18:49.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: CURRENT file: CURRENT 2024-04-15T22:18:49.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: IDENTITY file: IDENTITY 2024-04-15T22:18:49.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-15T22:18:49.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-15T22:18:49.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-15T22:18:49.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.error_if_exists: 0 2024-04-15T22:18:49.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.create_if_missing: 0 2024-04-15T22:18:49.196 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.paranoid_checks: 1 2024-04-15T22:18:49.196 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-15T22:18:49.196 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.env: 0x563a820c6860 2024-04-15T22:18:49.196 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.fs: Posix File System 2024-04-15T22:18:49.196 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.info_log: 0x563a84737dc0 2024-04-15T22:18:49.196 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_file_opening_threads: 16 2024-04-15T22:18:49.196 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.statistics: (nil) 2024-04-15T22:18:49.196 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.use_fsync: 0 2024-04-15T22:18:49.196 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_log_file_size: 0 2024-04-15T22:18:49.196 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-15T22:18:49.196 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-15T22:18:49.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.keep_log_file_num: 1000 2024-04-15T22:18:49.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.recycle_log_file_num: 0 2024-04-15T22:18:49.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.allow_fallocate: 1 2024-04-15T22:18:49.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.allow_mmap_reads: 0 2024-04-15T22:18:49.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.allow_mmap_writes: 0 2024-04-15T22:18:49.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.use_direct_reads: 0 2024-04-15T22:18:49.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-15T22:18:49.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.create_missing_column_families: 0 2024-04-15T22:18:49.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.db_log_dir: 2024-04-15T22:18:49.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-04-15T22:18:49.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-15T22:18:49.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-15T22:18:49.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-15T22:18:49.198 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-15T22:18:49.198 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-15T22:18:49.198 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-15T22:18:49.199 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.advise_random_on_open: 1 2024-04-15T22:18:49.199 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.db_write_buffer_size: 0 2024-04-15T22:18:49.199 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.write_buffer_manager: 0x563a848282d0 2024-04-15T22:18:49.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-15T22:18:49.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-15T22:18:49.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-15T22:18:49.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-15T22:18:49.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.rate_limiter: (nil) 2024-04-15T22:18:49.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-15T22:18:49.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.wal_recovery_mode: 2 2024-04-15T22:18:49.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.enable_thread_tracking: 0 2024-04-15T22:18:49.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.enable_pipelined_write: 0 2024-04-15T22:18:49.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.unordered_write: 0 2024-04-15T22:18:49.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-15T22:18:49.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-15T22:18:49.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-15T22:18:49.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-15T22:18:49.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.row_cache: None 2024-04-15T22:18:49.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.wal_filter: None 2024-04-15T22:18:49.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-15T22:18:49.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.allow_ingest_behind: 0 2024-04-15T22:18:49.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.preserve_deletes: 0 2024-04-15T22:18:49.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.two_write_queues: 0 2024-04-15T22:18:49.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.manual_wal_flush: 0 2024-04-15T22:18:49.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.atomic_flush: 0 2024-04-15T22:18:49.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-15T22:18:49.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-15T22:18:49.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-15T22:18:49.202 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.log_readahead_size: 0 2024-04-15T22:18:49.202 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-15T22:18:49.202 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.best_efforts_recovery: 0 2024-04-15T22:18:49.202 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-15T22:18:49.202 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-15T22:18:49.202 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.allow_data_in_errors: 0 2024-04-15T22:18:49.202 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.db_host_id: __hostname__ 2024-04-15T22:18:49.202 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_background_jobs: 2 2024-04-15T22:18:49.202 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_background_compactions: -1 2024-04-15T22:18:49.202 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_subcompactions: 1 2024-04-15T22:18:49.202 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-15T22:18:49.202 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_total_wal_size: 0 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_open_files: -1 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.bytes_per_sync: 0 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compaction_readahead_size: 0 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_background_flushes: -1 2024-04-15T22:18:49.203 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Compression algorithms supported: 2024-04-15T22:18:49.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-15T22:18:49.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: kZSTD supported: 0 2024-04-15T22:18:49.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: kXpressCompression supported: 0 2024-04-15T22:18:49.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: kLZ4HCCompression supported: 1 2024-04-15T22:18:49.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: kLZ4Compression supported: 1 2024-04-15T22:18:49.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: kBZip2Compression supported: 0 2024-04-15T22:18:49.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: kZlibCompression supported: 1 2024-04-15T22:18:49.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: kSnappyCompression supported: 1 2024-04-15T22:18:49.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-15T22:18:49.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-04-15T22:18:49.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-15T22:18:49.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-15T22:18:49.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.merge_operator: 2024-04-15T22:18:49.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compaction_filter: None 2024-04-15T22:18:49.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compaction_filter_factory: None 2024-04-15T22:18:49.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.sst_partitioner_factory: None 2024-04-15T22:18:49.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-15T22:18:49.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-15T22:18:49.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563a84703d00) 2024-04-15T22:18:49.205 INFO:journalctl@ceph.mon.c.smithi092.stdout: cache_index_and_filter_blocks: 1 2024-04-15T22:18:49.205 INFO:journalctl@ceph.mon.c.smithi092.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-15T22:18:49.206 INFO:journalctl@ceph.mon.c.smithi092.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-15T22:18:49.206 INFO:journalctl@ceph.mon.c.smithi092.stdout: pin_top_level_index_and_filter: 1 2024-04-15T22:18:49.206 INFO:journalctl@ceph.mon.c.smithi092.stdout: index_type: 0 2024-04-15T22:18:49.206 INFO:journalctl@ceph.mon.c.smithi092.stdout: data_block_index_type: 0 2024-04-15T22:18:49.206 INFO:journalctl@ceph.mon.c.smithi092.stdout: index_shortening: 1 2024-04-15T22:18:49.206 INFO:journalctl@ceph.mon.c.smithi092.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-15T22:18:49.206 INFO:journalctl@ceph.mon.c.smithi092.stdout: hash_index_allow_collision: 1 2024-04-15T22:18:49.206 INFO:journalctl@ceph.mon.c.smithi092.stdout: checksum: 1 2024-04-15T22:18:49.206 INFO:journalctl@ceph.mon.c.smithi092.stdout: no_block_cache: 0 2024-04-15T22:18:49.206 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_cache: 0x563a8476e170 2024-04-15T22:18:49.206 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_cache_name: BinnedLRUCache 2024-04-15T22:18:49.207 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_cache_options: 2024-04-15T22:18:49.207 INFO:journalctl@ceph.mon.c.smithi092.stdout: capacity : 536870912 2024-04-15T22:18:49.207 INFO:journalctl@ceph.mon.c.smithi092.stdout: num_shard_bits : 4 2024-04-15T22:18:49.207 INFO:journalctl@ceph.mon.c.smithi092.stdout: strict_capacity_limit : 0 2024-04-15T22:18:49.207 INFO:journalctl@ceph.mon.c.smithi092.stdout: high_pri_pool_ratio: 0.000 2024-04-15T22:18:49.207 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_cache_compressed: (nil) 2024-04-15T22:18:49.207 INFO:journalctl@ceph.mon.c.smithi092.stdout: persistent_cache: (nil) 2024-04-15T22:18:49.207 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_size: 4096 2024-04-15T22:18:49.207 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_size_deviation: 10 2024-04-15T22:18:49.207 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_restart_interval: 16 2024-04-15T22:18:49.207 INFO:journalctl@ceph.mon.c.smithi092.stdout: index_block_restart_interval: 1 2024-04-15T22:18:49.207 INFO:journalctl@ceph.mon.c.smithi092.stdout: metadata_block_size: 4096 2024-04-15T22:18:49.207 INFO:journalctl@ceph.mon.c.smithi092.stdout: partition_filters: 0 2024-04-15T22:18:49.208 INFO:journalctl@ceph.mon.c.smithi092.stdout: use_delta_encoding: 1 2024-04-15T22:18:49.208 INFO:journalctl@ceph.mon.c.smithi092.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-15T22:18:49.208 INFO:journalctl@ceph.mon.c.smithi092.stdout: whole_key_filtering: 1 2024-04-15T22:18:49.208 INFO:journalctl@ceph.mon.c.smithi092.stdout: verify_compression: 0 2024-04-15T22:18:49.208 INFO:journalctl@ceph.mon.c.smithi092.stdout: read_amp_bytes_per_bit: 0 2024-04-15T22:18:49.208 INFO:journalctl@ceph.mon.c.smithi092.stdout: format_version: 4 2024-04-15T22:18:49.208 INFO:journalctl@ceph.mon.c.smithi092.stdout: enable_index_compression: 1 2024-04-15T22:18:49.208 INFO:journalctl@ceph.mon.c.smithi092.stdout: block_align: 0 2024-04-15T22:18:49.208 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.write_buffer_size: 33554432 2024-04-15T22:18:49.208 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_write_buffer_number: 2 2024-04-15T22:18:49.208 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compression: NoCompression 2024-04-15T22:18:49.208 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.bottommost_compression: Disabled 2024-04-15T22:18:49.208 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.prefix_extractor: nullptr 2024-04-15T22:18:49.209 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-15T22:18:49.209 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.num_levels: 7 2024-04-15T22:18:49.209 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-15T22:18:49.209 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-15T22:18:49.209 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-15T22:18:49.209 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-15T22:18:49.209 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-15T22:18:49.209 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-15T22:18:49.209 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-15T22:18:49.209 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-15T22:18:49.209 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-15T22:18:49.209 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-15T22:18:49.209 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-15T22:18:49.210 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compression_opts.level: 32767 2024-04-15T22:18:49.210 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compression_opts.strategy: 0 2024-04-15T22:18:49.210 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-15T22:18:49.210 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-15T22:18:49.210 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-15T22:18:49.210 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compression_opts.enabled: false 2024-04-15T22:18:49.210 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-15T22:18:49.210 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-15T22:18:49.210 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-15T22:18:49.210 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.target_file_size_base: 67108864 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-15T22:18:49.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.arena_block_size: 4194304 2024-04-15T22:18:49.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-15T22:18:49.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-15T22:18:49.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-15T22:18:49.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.disable_auto_compactions: 0 2024-04-15T22:18:49.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-15T22:18:49.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-15T22:18:49.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-15T22:18:49.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-15T22:18:49.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-15T22:18:49.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-15T22:18:49.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-15T22:18:49.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-15T22:18:49.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-15T22:18:49.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-15T22:18:49.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.table_properties_collectors: 2024-04-15T22:18:49.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.inplace_update_support: 0 2024-04-15T22:18:49.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-15T22:18:49.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-15T22:18:49.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-15T22:18:49.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-15T22:18:49.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.bloom_locality: 0 2024-04-15T22:18:49.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.max_successive_merges: 0 2024-04-15T22:18:49.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-15T22:18:49.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.paranoid_file_checks: 0 2024-04-15T22:18:49.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.force_consistency_checks: 1 2024-04-15T22:18:49.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.report_bg_io_stats: 0 2024-04-15T22:18:49.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.ttl: 2592000 2024-04-15T22:18:49.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-15T22:18:49.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.enable_blob_files: false 2024-04-15T22:18:49.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.min_blob_size: 0 2024-04-15T22:18:49.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.blob_file_size: 268435456 2024-04-15T22:18:49.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-15T22:18:49.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-15T22:18:49.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-15T22:18:49.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-15T22:18:49.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-15T22:18:49.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-15T22:18:49.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713219528920322, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-15T22:18:49.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-15T22:18:49.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-15T22:18:49.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713219528920609, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713219528, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "3e945cea-68fb-48ab-bae1-95f9ffa869e9", "db_session_id": "FGJPYK049DHEN9Y0L0D8"}} 2024-04-15T22:18:49.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-15T22:18:49.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713219528920906, "job": 1, "event": "recovery_finished"} 2024-04-15T22:18:49.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-15T22:18:49.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x563a84754a80 2024-04-15T22:18:49.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: DB pointer 0x563a84764000 2024-04-15T22:18:49.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-15T22:18:49.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-15T22:18:49.216 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** DB Stats ** 2024-04-15T22:18:49.216 INFO:journalctl@ceph.mon.c.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T22:18:49.216 INFO:journalctl@ceph.mon.c.smithi092.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-15T22:18:49.216 INFO:journalctl@ceph.mon.c.smithi092.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T22:18:49.216 INFO:journalctl@ceph.mon.c.smithi092.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T22:18:49.216 INFO:journalctl@ceph.mon.c.smithi092.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-15T22:18:49.216 INFO:journalctl@ceph.mon.c.smithi092.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-15T22:18:49.216 INFO:journalctl@ceph.mon.c.smithi092.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T22:18:49.216 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-04-15T22:18:49.217 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** Compaction Stats [default] ** 2024-04-15T22:18:49.217 INFO:journalctl@ceph.mon.c.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T22:18:49.217 INFO:journalctl@ceph.mon.c.smithi092.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:18:49.217 INFO:journalctl@ceph.mon.c.smithi092.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:49.217 INFO:journalctl@ceph.mon.c.smithi092.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:49.217 INFO:journalctl@ceph.mon.c.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:49.217 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-04-15T22:18:49.217 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** Compaction Stats [default] ** 2024-04-15T22:18:49.217 INFO:journalctl@ceph.mon.c.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T22:18:49.217 INFO:journalctl@ceph.mon.c.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:18:49.217 INFO:journalctl@ceph.mon.c.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:49.217 INFO:journalctl@ceph.mon.c.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T22:18:49.218 INFO:journalctl@ceph.mon.c.smithi092.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T22:18:49.218 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T22:18:49.218 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T22:18:49.218 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T22:18:49.218 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T22:18:49.218 INFO:journalctl@ceph.mon.c.smithi092.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:18:49.218 INFO:journalctl@ceph.mon.c.smithi092.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:18:49.218 INFO:journalctl@ceph.mon.c.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T22:18:49.218 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-04-15T22:18:49.218 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T22:18:49.218 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-04-15T22:18:49.219 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** Compaction Stats [default] ** 2024-04-15T22:18:49.219 INFO:journalctl@ceph.mon.c.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T22:18:49.219 INFO:journalctl@ceph.mon.c.smithi092.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:18:49.219 INFO:journalctl@ceph.mon.c.smithi092.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:49.219 INFO:journalctl@ceph.mon.c.smithi092.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:49.219 INFO:journalctl@ceph.mon.c.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-15T22:18:49.219 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-04-15T22:18:49.219 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** Compaction Stats [default] ** 2024-04-15T22:18:49.219 INFO:journalctl@ceph.mon.c.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T22:18:49.219 INFO:journalctl@ceph.mon.c.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T22:18:49.219 INFO:journalctl@ceph.mon.c.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-04-15T22:18:49.220 INFO:journalctl@ceph.mon.c.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T22:18:49.220 INFO:journalctl@ceph.mon.c.smithi092.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T22:18:49.220 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T22:18:49.220 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T22:18:49.220 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T22:18:49.220 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T22:18:49.220 INFO:journalctl@ceph.mon.c.smithi092.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:18:49.220 INFO:journalctl@ceph.mon.c.smithi092.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T22:18:49.220 INFO:journalctl@ceph.mon.c.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T22:18:49.220 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-04-15T22:18:49.220 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T22:18:49.220 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-15T22:18:49.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: using public_addrv [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] 2024-04-15T22:18:49.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: starting mon.c rank -1 at public addrs [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] at bind addrs [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:18:49.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(???) e0 preinit fsid d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:18:49.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing).mds e1 new map 2024-04-15T22:18:49.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-15T22:18:49.221 INFO:journalctl@ceph.mon.c.smithi092.stdout: e1 2024-04-15T22:18:49.221 INFO:journalctl@ceph.mon.c.smithi092.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-15T22:18:49.221 INFO:journalctl@ceph.mon.c.smithi092.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-15T22:18:49.221 INFO:journalctl@ceph.mon.c.smithi092.stdout: legacy client fscid: -1 2024-04-15T22:18:49.221 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-04-15T22:18:49.221 INFO:journalctl@ceph.mon.c.smithi092.stdout: No filesystems configured 2024-04-15T22:18:49.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-15T22:18:49.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-15T22:18:49.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-15T22:18:49.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-15T22:18:49.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-15T22:18:49.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-15T22:18:49.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-15T22:18:49.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:18:49.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:18:49.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T22:18:49.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:49.223 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/172400840' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-15T22:18:49.223 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T22:18:49.223 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:49.223 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi092:172.21.15.92=a;smithi092:[v2:172.21.15.92:3301,v1:172.21.15.92:6790]=c;smithi176:172.21.15.176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:18:49.223 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: Saving service mon spec with placement smithi092:172.21.15.92=a;smithi092:[v2:172.21.15.92:3301,v1:172.21.15.92:6790]=c;smithi176:172.21.15.176=b;count:3 2024-04-15T22:18:49.223 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:49.223 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:18:49.223 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:18:49.223 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:18:49.223 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:49.223 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:18:49.223 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:18:49.223 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: Deploying daemon mon.c on smithi092 2024-04-15T22:18:49.224 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:49.224 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: from='client.? 172.21.15.176:0/1279738944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:18:49.224 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:49.224 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-04-15T22:18:49.224 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 ceph-mon[29727]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-15T22:18:49.224 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 bash[29677]: 69aa2c1b085d0d7f19b354ec43764b665296c7a116e1f949ce606ee1531567b1 2024-04-15T22:18:49.224 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:48 smithi092 systemd[1]: Started Ceph mon.c for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:18:54.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:18:54.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: mon.a calling monitor election 2024-04-15T22:18:54.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:54.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:54.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:54.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:54.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: mon.c calling monitor election 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: monmap e2: 2 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0]} 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: fsmap 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: mgrmap e13: y(active, since 36s) 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: overall HEALTH_OK 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:18:54.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:18:54.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:54.217 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:18:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:18:53] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:18:54.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:18:54.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: mon.a calling monitor election 2024-04-15T22:18:54.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:54.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:54.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:54.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: mon.c calling monitor election 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: monmap e2: 2 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],c=[v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0]} 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: fsmap 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: mgrmap e13: y(active, since 36s) 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: overall HEALTH_OK 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:54.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:54.219 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:18:54.219 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:18:54.219 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:54 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:18:54.564 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-15T22:18:54.564 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":2,"fsid":"d537646e-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:18:48.940181Z","created":"2024-04-15T22:17:18.281141Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3300","nonce":0},{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3301","nonce":0},{"type":"v1","addr":"172.21.15.92:6790","nonce":0}]},"addr":"172.21.15.92:6790/0","public_addr":"172.21.15.92:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-15T22:18:54.567 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 2 2024-04-15T22:18:55.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:55 smithi092 ceph-mon[24813]: Deploying daemon mon.b on smithi176 2024-04-15T22:18:55.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:55 smithi092 ceph-mon[24813]: from='client.? 172.21.15.176:0/2788549152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:18:55.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:55 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:55.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:55 smithi092 ceph-mon[29727]: Deploying daemon mon.b on smithi176 2024-04-15T22:18:55.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:55 smithi092 ceph-mon[29727]: from='client.? 172.21.15.176:0/2788549152' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:18:55.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:55 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:18:56.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:56 smithi092 ceph-mon[24813]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:56.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:56 smithi092 ceph-mon[29727]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:56.742 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-15T22:18:56.742 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T22:18:58.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:18:58 smithi092 ceph-mon[29727]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:18:58.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:18:58 smithi092 ceph-mon[24813]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:03.815 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-04-15T22:19:03.815 INFO:teuthology.orchestra.run.smithi176.stdout:{"epoch":3,"fsid":"d537646e-fb75-11ee-bc8f-c7b262605968","modified":"2024-04-15T22:18:58.803602Z","created":"2024-04-15T22:17:18.281141Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3300","nonce":0},{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:3301","nonce":0},{"type":"v1","addr":"172.21.15.92:6790","nonce":0}]},"addr":"172.21.15.92:6790/0","public_addr":"172.21.15.92:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-15T22:19:03.816 INFO:teuthology.orchestra.run.smithi176.stderr:dumped monmap epoch 3 2024-04-15T22:19:04.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:19:04.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: mon.c calling monitor election 2024-04-15T22:19:04.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: mon.a calling monitor election 2024-04-15T22:19:04.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:04.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='client.? 172.21.15.176:0/3427395772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: monmap e3: 3 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0]} 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: fsmap 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: mgrmap e13: y(active, since 46s) 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-15T22:19:04.103 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-15T22:19:04.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-15T22:19:04.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: mon.b (rank 2) addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] is down (out of quorum) 2024-04-15T22:19:04.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:04.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:04.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:04.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:04.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:04.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:04.104 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:19:03] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:19:04.104 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:19:03] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: mon.c calling monitor election 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: mon.a calling monitor election 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='client.? 172.21.15.176:0/3427395772' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:04.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: monmap e3: 3 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0]} 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: fsmap 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: mgrmap e13: y(active, since 46s) 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: mon.b (rank 2) addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] is down (out of quorum) 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:04.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:04.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:04.490 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-15T22:19:04.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph config generate-minimal-conf 2024-04-15T22:19:05.213 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:19:04] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:19:05.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-15T22:19:05.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: Updating smithi176:/etc/ceph/ceph.conf 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: Reconfiguring mon.a (unknown last config time)... 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: Reconfiguring daemon mon.a on smithi092 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-15T22:19:05.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: Updating smithi176:/etc/ceph/ceph.conf 2024-04-15T22:19:05.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:19:05.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: Reconfiguring mon.a (unknown last config time)... 2024-04-15T22:19:05.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:19:05.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:19:05.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:05.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: Reconfiguring daemon mon.a on smithi092 2024-04-15T22:19:05.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:05.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:05.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:19:05 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:19:05] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:19:06.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[24813]: mon.b calling monitor election 2024-04-15T22:19:06.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[24813]: mon.b calling monitor election 2024-04-15T22:19:06.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[24813]: mon.c calling monitor election 2024-04-15T22:19:06.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[24813]: mon.a calling monitor election 2024-04-15T22:19:06.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[24813]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-15T22:19:06.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[24813]: monmap e3: 3 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0]} 2024-04-15T22:19:06.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[24813]: fsmap 2024-04-15T22:19:06.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[24813]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T22:19:06.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[24813]: mgrmap e13: y(active, since 48s) 2024-04-15T22:19:06.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[24813]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-15T22:19:06.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[24813]: Cluster is now healthy 2024-04-15T22:19:06.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[24813]: overall HEALTH_OK 2024-04-15T22:19:06.606 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[29727]: mon.b calling monitor election 2024-04-15T22:19:06.607 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[29727]: mon.b calling monitor election 2024-04-15T22:19:06.607 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[29727]: mon.c calling monitor election 2024-04-15T22:19:06.607 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[29727]: mon.a calling monitor election 2024-04-15T22:19:06.607 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[29727]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-15T22:19:06.607 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[29727]: monmap e3: 3 mons at {a=[v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0],b=[v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0],c=[v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0]} 2024-04-15T22:19:06.607 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[29727]: fsmap 2024-04-15T22:19:06.607 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[29727]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T22:19:06.607 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[29727]: mgrmap e13: y(active, since 48s) 2024-04-15T22:19:06.607 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[29727]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-15T22:19:06.607 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[29727]: Cluster is now healthy 2024-04-15T22:19:06.607 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:06 smithi092 ceph-mon[29727]: overall HEALTH_OK 2024-04-15T22:19:07.353 INFO:teuthology.orchestra.run.smithi092.stdout:# minimal ceph.conf for d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:19:07.353 INFO:teuthology.orchestra.run.smithi092.stdout:[global] 2024-04-15T22:19:07.353 INFO:teuthology.orchestra.run.smithi092.stdout: fsid = d537646e-fb75-11ee-bc8f-c7b262605968 2024-04-15T22:19:07.353 INFO:teuthology.orchestra.run.smithi092.stdout: mon_host = [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] [v2:172.21.15.92:3301/0,v1:172.21.15.92:6790/0] 2024-04-15T22:19:07.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:19:07 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:19:07] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:19:07.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:07 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:07.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:07 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:07.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:07 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:07.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:07 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3265836416' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:07.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:07 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:19:07.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:07 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:07.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:07 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:07.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:07 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3265836416' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:07.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:07 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3265836416' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:08.183 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-15T22:19:08.183 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-15T22:19:08.183 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.conf 2024-04-15T22:19:08.216 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-15T22:19:08.216 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:19:08.284 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-15T22:19:08.284 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.conf 2024-04-15T22:19:08.323 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-15T22:19:08.323 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:19:08.395 INFO:tasks.cephadm:Adding mgr.y on smithi092 2024-04-15T22:19:08.395 INFO:tasks.cephadm:Adding mgr.x on smithi176 2024-04-15T22:19:08.395 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch apply mgr '2;smithi092=y;smithi176=x' 2024-04-15T22:19:08.883 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:08 smithi092 ceph-mon[24813]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:08.883 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:08.883 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:19:08 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:19:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:19:08.883 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:08 smithi092 ceph-mon[29727]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:08.883 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:08.976 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:08 smithi176 ceph-mon[25776]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:08.977 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:09.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:09 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:09.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:09 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:09.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:09 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:09.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:09 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:09.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:09 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:09.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:09 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:10.357 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled mgr update... 2024-04-15T22:19:10.917 DEBUG:teuthology.orchestra.run.smithi176:mgr.x> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mgr.x.service 2024-04-15T22:19:10.919 INFO:tasks.cephadm:Deploying OSDs... 2024-04-15T22:19:10.920 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-15T22:19:10.920 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T22:19:10.936 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T22:19:10.937 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2024-04-15T22:19:10.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:10 smithi176 ceph-mon[25776]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:10.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:10 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:10.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:10 smithi176 ceph-mon[25776]: Reconfiguring mon.c (monmap changed)... 2024-04-15T22:19:10.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:10 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:19:10.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:10 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:19:10.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:10 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:10.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:10 smithi176 ceph-mon[25776]: Reconfiguring daemon mon.c on smithi092 2024-04-15T22:19:10.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:10 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:10.993 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T22:19:10.993 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:19:10.994 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-15T22:19:10.994 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:19:10.994 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:19:10.994 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-04-15 22:19:07.272847754 +0000 2024-04-15T22:19:10.994 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-04-15 22:16:15.055005124 +0000 2024-04-15T22:19:10.994 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-04-15 22:16:15.055005124 +0000 2024-04-15T22:19:10.994 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-04-15 22:16:15.055005124 +0000 2024-04-15T22:19:10.994 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T22:19:11.033 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[24813]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:11.033 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:11.033 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[24813]: Reconfiguring mon.c (monmap changed)... 2024-04-15T22:19:11.033 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:19:11.033 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:19:11.034 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:11.034 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[24813]: Reconfiguring daemon mon.c on smithi092 2024-04-15T22:19:11.034 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:11.034 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[29727]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:11.034 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:11.034 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[29727]: Reconfiguring mon.c (monmap changed)... 2024-04-15T22:19:11.034 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:19:11.034 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:19:11.034 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:11.034 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[29727]: Reconfiguring daemon mon.c on smithi092 2024-04-15T22:19:11.035 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:10 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:11.065 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-04-15T22:19:11.066 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-04-15T22:19:11.066 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000219643 s, 2.3 MB/s 2024-04-15T22:19:11.067 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T22:19:11.124 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2024-04-15T22:19:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T22:19:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:19:11.180 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-15T22:19:11.180 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:19:11.180 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:19:11.180 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-04-15 22:19:07.272847754 +0000 2024-04-15T22:19:11.180 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-04-15 22:16:15.412995061 +0000 2024-04-15T22:19:11.180 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-04-15 22:16:15.412995061 +0000 2024-04-15T22:19:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-04-15 22:16:15.412995061 +0000 2024-04-15T22:19:11.181 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T22:19:11.247 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-04-15T22:19:11.247 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-04-15T22:19:11.247 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000272159 s, 1.9 MB/s 2024-04-15T22:19:11.248 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T22:19:11.305 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2024-04-15T22:19:11.361 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T22:19:11.362 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:19:11.362 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-15T22:19:11.362 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:19:11.362 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:19:11.362 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-04-15 22:19:07.272847754 +0000 2024-04-15T22:19:11.362 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-04-15 22:16:15.755985420 +0000 2024-04-15T22:19:11.362 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-04-15 22:16:15.755985420 +0000 2024-04-15T22:19:11.362 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-04-15 22:16:15.755985420 +0000 2024-04-15T22:19:11.362 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T22:19:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-04-15T22:19:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-04-15T22:19:11.429 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000219344 s, 2.3 MB/s 2024-04-15T22:19:11.430 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T22:19:11.486 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2024-04-15T22:19:11.542 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T22:19:11.542 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:19:11.542 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-15T22:19:11.542 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:19:11.542 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:19:11.542 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-04-15 22:19:07.272847754 +0000 2024-04-15T22:19:11.542 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-04-15 22:16:16.110975441 +0000 2024-04-15T22:19:11.542 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-04-15 22:16:16.110975441 +0000 2024-04-15T22:19:11.543 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-04-15 22:16:16.110975441 +0000 2024-04-15T22:19:11.543 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T22:19:11.610 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-04-15T22:19:11.610 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-04-15T22:19:11.610 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000195504 s, 2.6 MB/s 2024-04-15T22:19:11.611 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T22:19:11.668 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-15T22:19:11.668 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T22:19:11.685 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T22:19:11.686 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-04-15T22:19:11.741 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T22:19:11.742 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:19:11.742 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-04-15T22:19:11.742 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:19:11.742 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:19:11.742 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-04-15 22:19:06.552623720 +0000 2024-04-15T22:19:11.742 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-04-15 22:16:15.066048500 +0000 2024-04-15T22:19:11.742 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-04-15 22:16:15.066048500 +0000 2024-04-15T22:19:11.742 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-04-15 22:16:15.066048500 +0000 2024-04-15T22:19:11.742 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T22:19:11.809 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-04-15T22:19:11.810 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-04-15T22:19:11.810 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000191435 s, 2.7 MB/s 2024-04-15T22:19:11.811 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T22:19:11.867 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-04-15T22:19:11.922 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T22:19:11.923 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:19:11.923 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-15T22:19:11.923 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:19:11.923 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:19:11.923 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-04-15 22:19:06.552623720 +0000 2024-04-15T22:19:11.923 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-04-15 22:16:15.414045050 +0000 2024-04-15T22:19:11.923 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-04-15 22:16:15.414045050 +0000 2024-04-15T22:19:11.923 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-04-15 22:16:15.414045050 +0000 2024-04-15T22:19:11.923 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T22:19:11.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:11 smithi176 ceph-mon[25776]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi092=y;smithi176=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:19:11.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:11 smithi176 ceph-mon[25776]: Saving service mgr spec with placement smithi092=y;smithi176=x;count:2 2024-04-15T22:19:11.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:11 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:11.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:11 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:19:11.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:11 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:19:11.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:11 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:11.990 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-04-15T22:19:11.990 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-04-15T22:19:11.990 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000221406 s, 2.3 MB/s 2024-04-15T22:19:11.992 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T22:19:12.048 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-04-15T22:19:12.103 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T22:19:12.103 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:19:12.103 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-04-15T22:19:12.103 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:19:12.104 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:19:12.104 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-04-15 22:19:06.552623720 +0000 2024-04-15T22:19:12.104 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-04-15 22:16:15.767041550 +0000 2024-04-15T22:19:12.104 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-04-15 22:16:15.767041550 +0000 2024-04-15T22:19:12.104 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-04-15 22:16:15.767041550 +0000 2024-04-15T22:19:12.104 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T22:19:12.170 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-04-15T22:19:12.171 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-04-15T22:19:12.171 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000196703 s, 2.6 MB/s 2024-04-15T22:19:12.172 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T22:19:12.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:11 smithi092 ceph-mon[24813]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi092=y;smithi176=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:19:12.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:11 smithi092 ceph-mon[24813]: Saving service mgr spec with placement smithi092=y;smithi176=x;count:2 2024-04-15T22:19:12.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:11 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:12.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:11 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:19:12.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:11 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:19:12.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:11 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:12.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:11 smithi092 ceph-mon[29727]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi092=y;smithi176=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:19:12.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:11 smithi092 ceph-mon[29727]: Saving service mgr spec with placement smithi092=y;smithi176=x;count:2 2024-04-15T22:19:12.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:11 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:12.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:11 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T22:19:12.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:11 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T22:19:12.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:11 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:12.228 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-04-15T22:19:12.283 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T22:19:12.283 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T22:19:12.283 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-15T22:19:12.283 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T22:19:12.283 INFO:teuthology.orchestra.run.smithi176.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T22:19:12.283 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-04-15 22:19:06.552623720 +0000 2024-04-15T22:19:12.283 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-04-15 22:16:16.124038011 +0000 2024-04-15T22:19:12.283 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-04-15 22:16:16.124038011 +0000 2024-04-15T22:19:12.283 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: 2024-04-15 22:16:16.124038011 +0000 2024-04-15T22:19:12.284 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T22:19:12.349 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-04-15T22:19:12.349 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-04-15T22:19:12.349 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.00016755 s, 3.1 MB/s 2024-04-15T22:19:12.351 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T22:19:12.407 INFO:tasks.cephadm:Deploying osd.0 on smithi092 with /dev/vg_nvme/lv_4... 2024-04-15T22:19:12.407 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-15T22:19:12.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:12 smithi176 ceph-mon[25776]: Reconfiguring mon.b (monmap changed)... 2024-04-15T22:19:12.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:12 smithi176 ceph-mon[25776]: Reconfiguring daemon mon.b on smithi176 2024-04-15T22:19:12.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:12 smithi176 ceph-mon[25776]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:12.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:12.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:12.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:12.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:12.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:12.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:19:12.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-15T22:19:12.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:19:12.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:13.003 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[29727]: Reconfiguring mon.b (monmap changed)... 2024-04-15T22:19:13.003 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[29727]: Reconfiguring daemon mon.b on smithi176 2024-04-15T22:19:13.003 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[29727]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:13.003 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:13.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:13.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:13.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:13.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:13.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:19:13.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-15T22:19:13.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:19:13.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:13.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[24813]: Reconfiguring mon.b (monmap changed)... 2024-04-15T22:19:13.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[24813]: Reconfiguring daemon mon.b on smithi176 2024-04-15T22:19:13.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[24813]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:13.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:13.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:13.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:13.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:13.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:13.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:19:13.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-15T22:19:13.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:19:13.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:13.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:13 smithi176 ceph-mon[25776]: Deploying daemon mgr.x on smithi176 2024-04-15T22:19:14.003 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:13 smithi092 ceph-mon[24813]: Deploying daemon mgr.x on smithi176 2024-04-15T22:19:14.003 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:13 smithi092 ceph-mon[29727]: Deploying daemon mgr.x on smithi176 2024-04-15T22:19:14.477 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-15T22:19:14.477 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-15T22:19:14.477 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-04-15T22:19:14.477 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-04-15T22:19:14.477 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113866 s, 921 MB/s 2024-04-15T22:19:14.477 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-04-15T22:19:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:14 smithi176 ceph-mon[25776]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:14 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:14 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:14 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:14 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:14 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:14 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:19:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:14 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:19:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:14 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:15.088 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[24813]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:15.088 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:15.088 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:15.088 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:15.088 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:15.088 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:15.088 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:19:15.088 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:19:15.088 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:15.089 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[29727]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:15.089 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:15.089 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:15.089 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:15.089 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:15.089 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:15.089 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T22:19:15.089 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T22:19:15.089 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:14 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:15.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi092:vg_nvme/lv_4 2024-04-15T22:19:15.729 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:15.692+0000 7f190e9a5000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T22:19:16.022 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:15 smithi176 ceph-mon[25776]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:19:16.022 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:15 smithi176 ceph-mon[25776]: Reconfiguring mgr.y (unknown last config time)... 2024-04-15T22:19:16.023 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:15 smithi176 ceph-mon[25776]: Reconfiguring daemon mgr.y on smithi092 2024-04-15T22:19:16.023 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:15.782+0000 7f190e9a5000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T22:19:16.023 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:15.867+0000 7f190e9a5000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T22:19:16.052 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:15 smithi092 ceph-mon[24813]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:19:16.052 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:15 smithi092 ceph-mon[24813]: Reconfiguring mgr.y (unknown last config time)... 2024-04-15T22:19:16.052 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:15 smithi092 ceph-mon[24813]: Reconfiguring daemon mgr.y on smithi092 2024-04-15T22:19:16.052 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:15 smithi092 ceph-mon[29727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:19:16.052 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:15 smithi092 ceph-mon[29727]: Reconfiguring mgr.y (unknown last config time)... 2024-04-15T22:19:16.052 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:15 smithi092 ceph-mon[29727]: Reconfiguring daemon mgr.y on smithi092 2024-04-15T22:19:16.309 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:16 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:16.021+0000 7f190e9a5000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T22:19:16.309 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:16 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:16.103+0000 7f190e9a5000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T22:19:16.309 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:16 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:16.219+0000 7f190e9a5000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T22:19:16.309 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:16 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:16.307+0000 7f190e9a5000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T22:19:16.568 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:16 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:16.502+0000 7f190e9a5000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T22:19:16.938 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:16 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:16.603+0000 7f190e9a5000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T22:19:16.939 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:16 smithi176 ceph-mon[25776]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:16.939 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:16 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:16.939 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:16 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:16.939 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:16 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:16.939 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:16 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:16.939 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:16 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:17.175 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:16 smithi092 ceph-mon[24813]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:17.175 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:16 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:17.175 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:16 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:17.175 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:16 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:17.175 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:16 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:17.175 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:16 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:17.175 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:16 smithi092 ceph-mon[29727]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:17.175 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:16 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:17.176 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:16 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:17.176 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:16 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:17.176 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:16 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:17.176 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:16 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:17.715 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:19:17 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:19:17] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:19:17.864 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:17 smithi176 ceph-mon[25776]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:19:17.864 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:17 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:17.864 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:17 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:19:17.864 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:17 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:19:17.864 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:17 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:17.864 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:17 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:17.732+0000 7f190e9a5000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T22:19:17.864 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:17 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:17.862+0000 7f190e9a5000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T22:19:18.014 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:17 smithi092 ceph-mon[24813]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:19:18.014 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:17 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:18.014 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:17 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:19:18.014 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:17 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:19:18.014 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:17 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:18.014 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:17 smithi092 ceph-mon[29727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:19:18.014 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:17 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:18.015 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:17 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:19:18.015 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:17 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:19:18.015 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:17 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:18.150 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:18.063+0000 7f190e9a5000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T22:19:18.150 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:18.149+0000 7f190e9a5000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T22:19:18.305 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:19:18] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:19:18.479 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:18.284+0000 7f190e9a5000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T22:19:18.909 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:18.603+0000 7f190e9a5000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T22:19:18.909 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:18.823+0000 7f190e9a5000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T22:19:18.909 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:18.907+0000 7f190e9a5000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T22:19:18.909 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-mon[25776]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:18.909 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:19:18.909 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:19:18.910 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:18.910 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:18.910 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:18.910 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:18.910 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:18.910 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:18.910 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:19.134 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[24813]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:19.134 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:19:19.134 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:19:19.134 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:19.134 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:19.134 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:19.135 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:19.135 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:19.135 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:19.135 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:19.135 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[29727]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:19.135 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:19:19.135 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:19:19.135 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:19.135 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:19.135 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:19.135 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:19.136 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:19.136 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:19.136 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:19.229 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:19 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:19.077+0000 7f190e9a5000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T22:19:20.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:19 smithi092 ceph-mon[24813]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:19:20.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:19 smithi092 ceph-mon[29727]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:19:20.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:19 smithi176 ceph-mon[25776]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:19:20.229 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:19 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:19.988+0000 7f190e9a5000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T22:19:20.729 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:20.280+0000 7f190e9a5000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T22:19:20.729 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:20.374+0000 7f190e9a5000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T22:19:20.729 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:19:20.523+0000 7f190e9a5000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T22:19:21.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[24813]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:21.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/287686687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ae72c2d-f0aa-4631-9599-82b8a4771c4f"}]: dispatch 2024-04-15T22:19:21.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ae72c2d-f0aa-4631-9599-82b8a4771c4f"}]: dispatch 2024-04-15T22:19:21.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ae72c2d-f0aa-4631-9599-82b8a4771c4f"}]': finished 2024-04-15T22:19:21.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[24813]: osdmap e5: 1 total, 0 up, 1 in 2024-04-15T22:19:21.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:21.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[24813]: Standby manager daemon x started 2024-04-15T22:19:21.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[24813]: from='mgr.? 172.21.15.176:0/2511063161' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T22:19:21.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[24813]: from='mgr.? 172.21.15.176:0/2511063161' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T22:19:21.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[24813]: from='mgr.? 172.21.15.176:0/2511063161' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T22:19:21.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[24813]: from='mgr.? 172.21.15.176:0/2511063161' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T22:19:21.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/627602308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:19:21.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[29727]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:21.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/287686687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ae72c2d-f0aa-4631-9599-82b8a4771c4f"}]: dispatch 2024-04-15T22:19:21.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[29727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ae72c2d-f0aa-4631-9599-82b8a4771c4f"}]: dispatch 2024-04-15T22:19:21.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[29727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ae72c2d-f0aa-4631-9599-82b8a4771c4f"}]': finished 2024-04-15T22:19:21.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[29727]: osdmap e5: 1 total, 0 up, 1 in 2024-04-15T22:19:21.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:21.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[29727]: Standby manager daemon x started 2024-04-15T22:19:21.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[29727]: from='mgr.? 172.21.15.176:0/2511063161' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T22:19:21.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[29727]: from='mgr.? 172.21.15.176:0/2511063161' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T22:19:21.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[29727]: from='mgr.? 172.21.15.176:0/2511063161' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T22:19:21.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[29727]: from='mgr.? 172.21.15.176:0/2511063161' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T22:19:21.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:20 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/627602308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:19:21.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-mon[25776]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:21.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/287686687' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ae72c2d-f0aa-4631-9599-82b8a4771c4f"}]: dispatch 2024-04-15T22:19:21.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-mon[25776]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8ae72c2d-f0aa-4631-9599-82b8a4771c4f"}]: dispatch 2024-04-15T22:19:21.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-mon[25776]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8ae72c2d-f0aa-4631-9599-82b8a4771c4f"}]': finished 2024-04-15T22:19:21.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-mon[25776]: osdmap e5: 1 total, 0 up, 1 in 2024-04-15T22:19:21.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:21.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-mon[25776]: Standby manager daemon x started 2024-04-15T22:19:21.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-mon[25776]: from='mgr.? 172.21.15.176:0/2511063161' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T22:19:21.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-mon[25776]: from='mgr.? 172.21.15.176:0/2511063161' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T22:19:21.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-mon[25776]: from='mgr.? 172.21.15.176:0/2511063161' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T22:19:21.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-mon[25776]: from='mgr.? 172.21.15.176:0/2511063161' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T22:19:21.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:20 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/627602308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:19:22.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:22 smithi092 ceph-mon[24813]: mgrmap e14: y(active, since 63s), standbys: x 2024-04-15T22:19:22.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:22 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T22:19:22.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:22 smithi092 ceph-mon[24813]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:22.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:22 smithi092 ceph-mon[29727]: mgrmap e14: y(active, since 63s), standbys: x 2024-04-15T22:19:22.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:22 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T22:19:22.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:22 smithi092 ceph-mon[29727]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:22.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:22 smithi176 ceph-mon[25776]: mgrmap e14: y(active, since 63s), standbys: x 2024-04-15T22:19:22.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:22 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T22:19:22.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:22 smithi176 ceph-mon[25776]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:24.831 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:24 smithi092 ceph-mon[24813]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:24.832 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:24 smithi092 ceph-mon[29727]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:24.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:24 smithi176 ceph-mon[25776]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:26.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:26 smithi092 ceph-mon[29727]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:26.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:26 smithi092 ceph-mon[24813]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:26.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:26 smithi176 ceph-mon[25776]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:27.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:27 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-15T22:19:27.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:27 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:28.083 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-15T22:19:28.083 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:28.083 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:19:27 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:19:27] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:19:28.084 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:27 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-15T22:19:28.084 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:27 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:28.960 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:28 smithi092 ceph-mon[24813]: Deploying daemon osd.0 on smithi092 2024-04-15T22:19:28.960 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:28 smithi092 ceph-mon[24813]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:28.960 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:28.961 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:28.961 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:28 smithi092 ceph-mon[29727]: Deploying daemon osd.0 on smithi092 2024-04-15T22:19:28.961 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:28 smithi092 ceph-mon[29727]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:28.961 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:28.961 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:28.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:28 smithi176 ceph-mon[25776]: Deploying daemon osd.0 on smithi092 2024-04-15T22:19:28.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:28 smithi176 ceph-mon[25776]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:28.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:28.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:30.193 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:30 smithi092 ceph-mon[24813]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:30.193 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:30 smithi092 ceph-mon[29727]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:30.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:30 smithi176 ceph-mon[25776]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:32.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:32 smithi092 ceph-mon[24813]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:32.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:32 smithi092 ceph-mon[29727]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:32.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:32 smithi176 ceph-mon[25776]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:34.922 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:34 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:34.922 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:34 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:34.922 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:34 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:34.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:34 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:34.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:34 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:34.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:34 smithi092 ceph-mon[24813]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:34.923 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:34 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:34.923 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:34 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:34.923 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:34 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:34.923 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:34 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:34.923 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:34 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:34.923 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:34 smithi092 ceph-mon[29727]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:34.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:34 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:34.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:34 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:34.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:34 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:34.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:34 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:34.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:34 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:34.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:34 smithi176 ceph-mon[25776]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:36.287 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 0 on host 'smithi092' 2024-04-15T22:19:36.800 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:36 smithi176 ceph-mon[25776]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:36.800 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:36.800 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:36.800 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:36.800 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:36.800 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:19:36.800 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:36.800 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:36.800 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:36 smithi176 ceph-mon[25776]: from='osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-15T22:19:36.916 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[29727]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:36.916 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:36.916 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:36.916 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:36.916 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:36.916 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:19:36.916 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:36.916 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:36.916 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[29727]: from='osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-15T22:19:36.916 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[24813]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:36.917 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:36.917 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:36.917 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:36.917 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:36.917 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:19:36.917 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:36.917 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:36.917 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:36 smithi092 ceph-mon[24813]: from='osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-15T22:19:36.937 DEBUG:teuthology.orchestra.run.smithi092:osd.0> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.0.service 2024-04-15T22:19:36.940 INFO:tasks.cephadm:Deploying osd.1 on smithi092 with /dev/vg_nvme/lv_3... 2024-04-15T22:19:36.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-15T22:19:37.214 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:19:37] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:19:37.928 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:37 smithi176 ceph-mon[25776]: Adjusting osd_memory_target on smithi092 to 15040M 2024-04-15T22:19:37.928 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:37 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:37.928 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:37 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:37.928 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:37 smithi176 ceph-mon[25776]: from='osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-15T22:19:37.928 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:37 smithi176 ceph-mon[25776]: osdmap e6: 1 total, 0 up, 1 in 2024-04-15T22:19:37.929 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:37 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:37.929 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:37 smithi176 ceph-mon[25776]: from='osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:19:37.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[29727]: Adjusting osd_memory_target on smithi092 to 15040M 2024-04-15T22:19:37.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:37.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:37.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[29727]: from='osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-15T22:19:37.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[29727]: osdmap e6: 1 total, 0 up, 1 in 2024-04-15T22:19:37.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:37.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[29727]: from='osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:19:37.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[24813]: Adjusting osd_memory_target on smithi092 to 15040M 2024-04-15T22:19:37.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:37.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:37.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[24813]: from='osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-15T22:19:37.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[24813]: osdmap e6: 1 total, 0 up, 1 in 2024-04-15T22:19:37.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:37.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:37 smithi092 ceph-mon[24813]: from='osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:19:38.634 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:19:38 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0[33002]: 2024-04-15T22:19:38.343+0000 7f4ebe7e3700 -1 osd.0 0 waiting for initial osdmap 2024-04-15T22:19:38.634 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:19:38 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0[33002]: 2024-04-15T22:19:38.346+0000 7f4eba17c700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:19:38.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:38 smithi092 ceph-mon[29727]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:38.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:38 smithi092 ceph-mon[29727]: from='osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-15T22:19:38.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:38 smithi092 ceph-mon[29727]: osdmap e7: 1 total, 0 up, 1 in 2024-04-15T22:19:38.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:38 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:38.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:38 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:38.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:38 smithi092 ceph-mon[24813]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:38.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:38 smithi092 ceph-mon[24813]: from='osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-15T22:19:38.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:38 smithi092 ceph-mon[24813]: osdmap e7: 1 total, 0 up, 1 in 2024-04-15T22:19:38.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:38.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:38.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:38 smithi176 ceph-mon[25776]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:38.981 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:38 smithi176 ceph-mon[25776]: from='osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-15T22:19:38.981 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:38 smithi176 ceph-mon[25776]: osdmap e7: 1 total, 0 up, 1 in 2024-04-15T22:19:38.981 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:38 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:38.981 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:38 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:39.012 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-15T22:19:39.012 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-15T22:19:39.013 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-04-15T22:19:39.013 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-04-15T22:19:39.013 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128125 s, 818 MB/s 2024-04-15T22:19:39.013 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 2024-04-15T22:19:39.013 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-04-15T22:19:39.695 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi092:vg_nvme/lv_3 2024-04-15T22:19:40.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[24813]: purged_snaps scrub starts 2024-04-15T22:19:40.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[24813]: purged_snaps scrub ok 2024-04-15T22:19:40.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:40.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[24813]: osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669] boot 2024-04-15T22:19:40.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[24813]: osdmap e8: 1 total, 1 up, 1 in 2024-04-15T22:19:40.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:40.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[24813]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:40.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[29727]: purged_snaps scrub starts 2024-04-15T22:19:40.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[29727]: purged_snaps scrub ok 2024-04-15T22:19:40.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:40.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[29727]: osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669] boot 2024-04-15T22:19:40.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[29727]: osdmap e8: 1 total, 1 up, 1 in 2024-04-15T22:19:40.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:40.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:40 smithi092 ceph-mon[29727]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:40.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:40 smithi176 ceph-mon[25776]: purged_snaps scrub starts 2024-04-15T22:19:40.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:40 smithi176 ceph-mon[25776]: purged_snaps scrub ok 2024-04-15T22:19:40.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:40 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:40.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:40 smithi176 ceph-mon[25776]: osd.0 [v2:172.21.15.92:6802/3827592669,v1:172.21.15.92:6803/3827592669] boot 2024-04-15T22:19:40.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:40 smithi176 ceph-mon[25776]: osdmap e8: 1 total, 1 up, 1 in 2024-04-15T22:19:40.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:40 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:19:40.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:40 smithi176 ceph-mon[25776]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T22:19:41.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:41 smithi092 ceph-mon[24813]: osdmap e9: 1 total, 1 up, 1 in 2024-04-15T22:19:41.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:41 smithi092 ceph-mon[29727]: osdmap e9: 1 total, 1 up, 1 in 2024-04-15T22:19:41.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:41 smithi176 ceph-mon[25776]: osdmap e9: 1 total, 1 up, 1 in 2024-04-15T22:19:42.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:42 smithi092 ceph-mon[24813]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:19:42.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:42 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:19:42.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:42 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:19:42.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:42 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:42.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:42 smithi092 ceph-mon[24813]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:42.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:42 smithi092 ceph-mon[29727]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:19:42.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:42 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:19:42.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:42 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:19:42.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:42 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:42.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:42 smithi092 ceph-mon[29727]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:42.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:42 smithi176 ceph-mon[25776]: from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:19:42.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:42 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:19:42.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:42 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:19:42.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:42 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:42.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:42 smithi176 ceph-mon[25776]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:44.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[24813]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:44.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3978808041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa81f8b4-e851-4b06-bbbc-c5433ec9617a"}]: dispatch 2024-04-15T22:19:44.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa81f8b4-e851-4b06-bbbc-c5433ec9617a"}]: dispatch 2024-04-15T22:19:44.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa81f8b4-e851-4b06-bbbc-c5433ec9617a"}]': finished 2024-04-15T22:19:44.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[24813]: osdmap e10: 2 total, 1 up, 2 in 2024-04-15T22:19:44.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:19:44.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2464038917' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:19:44.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:44.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:44.965 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:19:44] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:19:44.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[29727]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:44.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3978808041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa81f8b4-e851-4b06-bbbc-c5433ec9617a"}]: dispatch 2024-04-15T22:19:44.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[29727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa81f8b4-e851-4b06-bbbc-c5433ec9617a"}]: dispatch 2024-04-15T22:19:44.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[29727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa81f8b4-e851-4b06-bbbc-c5433ec9617a"}]': finished 2024-04-15T22:19:44.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[29727]: osdmap e10: 2 total, 1 up, 2 in 2024-04-15T22:19:44.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:19:44.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2464038917' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:19:44.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:44.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:44 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:44.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:44 smithi176 ceph-mon[25776]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:44.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:44 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3978808041' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa81f8b4-e851-4b06-bbbc-c5433ec9617a"}]: dispatch 2024-04-15T22:19:44.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:44 smithi176 ceph-mon[25776]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa81f8b4-e851-4b06-bbbc-c5433ec9617a"}]: dispatch 2024-04-15T22:19:44.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:44 smithi176 ceph-mon[25776]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa81f8b4-e851-4b06-bbbc-c5433ec9617a"}]': finished 2024-04-15T22:19:44.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:44 smithi176 ceph-mon[25776]: osdmap e10: 2 total, 1 up, 2 in 2024-04-15T22:19:44.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:44 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:19:44.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:44 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2464038917' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:19:44.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:44 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:44.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:44 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:45.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:45 smithi176 ceph-mon[25776]: Detected new or changed devices on smithi092 2024-04-15T22:19:46.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:45 smithi092 ceph-mon[24813]: Detected new or changed devices on smithi092 2024-04-15T22:19:46.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:45 smithi092 ceph-mon[29727]: Detected new or changed devices on smithi092 2024-04-15T22:19:46.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:46 smithi092 ceph-mon[24813]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:46.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:46 smithi092 ceph-mon[29727]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:46.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:46 smithi176 ceph-mon[25776]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:48.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:48 smithi176 ceph-mon[25776]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:49.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:48 smithi092 ceph-mon[24813]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:49.023 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:48 smithi092 ceph-mon[29727]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:50.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:50 smithi092 ceph-mon[29727]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:50.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:50 smithi092 ceph-mon[24813]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:50.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:50 smithi176 ceph-mon[25776]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:51.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:51 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-15T22:19:51.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:51 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:51.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:51 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-15T22:19:51.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:51 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:51.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:51 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-15T22:19:51.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:51 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:52.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:52 smithi176 ceph-mon[25776]: Deploying daemon osd.1 on smithi092 2024-04-15T22:19:52.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:52 smithi176 ceph-mon[25776]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:52.538 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:52 smithi092 ceph-mon[29727]: Deploying daemon osd.1 on smithi092 2024-04-15T22:19:52.538 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:52 smithi092 ceph-mon[29727]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:52.539 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:52 smithi092 ceph-mon[24813]: Deploying daemon osd.1 on smithi092 2024-04-15T22:19:52.539 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:52 smithi092 ceph-mon[24813]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:54.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:54 smithi092 ceph-mon[24813]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:54.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:54 smithi092 ceph-mon[29727]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:54.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:54 smithi176 ceph-mon[25776]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:56.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:56 smithi092 ceph-mon[24813]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:56.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:56 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:56.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:56 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:56.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:56 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:56.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:56 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:56.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:56 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:56.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:56 smithi092 ceph-mon[29727]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:56.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:56 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:56.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:56 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:56.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:56 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:56.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:56 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:56.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:56 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:56.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:56 smithi176 ceph-mon[25776]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:56.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:56 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:56.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:56 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:19:56.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:56 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:19:56.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:56 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:19:56.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:56 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:57.526 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:19:57 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:19:57] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:19:58.340 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:58 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:58.340 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:58 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:58.340 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:58 smithi176 ceph-mon[25776]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:58.446 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:58 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:58.446 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:58 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:58.446 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:58 smithi092 ceph-mon[29727]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:58.447 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:58 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:58.447 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:58 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:19:58.447 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:58 smithi092 ceph-mon[24813]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:19:58.843 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 1 on host 'smithi092' 2024-04-15T22:19:59.479 DEBUG:teuthology.orchestra.run.smithi092:osd.1> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.1.service 2024-04-15T22:19:59.482 INFO:tasks.cephadm:Deploying osd.2 on smithi092 with /dev/vg_nvme/lv_2... 2024-04-15T22:19:59.482 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-15T22:20:00.080 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:00.080 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:00.080 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:00.080 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:00.081 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:00.081 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:00.081 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[29727]: Adjusting osd_memory_target on smithi092 to 7520M 2024-04-15T22:20:00.081 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:00.081 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:00.081 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[29727]: from='osd.1 [v2:172.21.15.92:6810/1540392933,v1:172.21.15.92:6811/1540392933]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T22:20:00.081 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[29727]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T22:20:00.081 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[29727]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:20:00.081 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:00.081 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:00.081 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:00.081 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:00.081 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:00.082 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:00.082 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[24813]: Adjusting osd_memory_target on smithi092 to 7520M 2024-04-15T22:20:00.082 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:00.082 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:00.082 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[24813]: from='osd.1 [v2:172.21.15.92:6810/1540392933,v1:172.21.15.92:6811/1540392933]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T22:20:00.082 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[24813]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T22:20:00.082 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:19:59 smithi092 ceph-mon[24813]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:20:00.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:59 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:00.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:59 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:00.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:59 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:00.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:59 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:00.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:59 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:00.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:59 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:00.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:59 smithi176 ceph-mon[25776]: Adjusting osd_memory_target on smithi092 to 7520M 2024-04-15T22:20:00.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:59 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:00.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:59 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:00.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:59 smithi176 ceph-mon[25776]: from='osd.1 [v2:172.21.15.92:6810/1540392933,v1:172.21.15.92:6811/1540392933]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T22:20:00.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:59 smithi176 ceph-mon[25776]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T22:20:00.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:19:59 smithi176 ceph-mon[25776]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:20:01.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:00 smithi092 ceph-mon[29727]: overall HEALTH_OK 2024-04-15T22:20:01.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:00 smithi092 ceph-mon[29727]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-15T22:20:01.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:00 smithi092 ceph-mon[29727]: osdmap e11: 2 total, 1 up, 2 in 2024-04-15T22:20:01.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:00 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:20:01.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:00 smithi092 ceph-mon[29727]: from='osd.1 [v2:172.21.15.92:6810/1540392933,v1:172.21.15.92:6811/1540392933]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:01.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:00 smithi092 ceph-mon[29727]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:01.093 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:00 smithi092 ceph-mon[24813]: overall HEALTH_OK 2024-04-15T22:20:01.093 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:00 smithi092 ceph-mon[24813]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-15T22:20:01.093 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:00 smithi092 ceph-mon[24813]: osdmap e11: 2 total, 1 up, 2 in 2024-04-15T22:20:01.093 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:00 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:20:01.093 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:00 smithi092 ceph-mon[24813]: from='osd.1 [v2:172.21.15.92:6810/1540392933,v1:172.21.15.92:6811/1540392933]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:01.093 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:00 smithi092 ceph-mon[24813]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:01.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:00 smithi176 ceph-mon[25776]: overall HEALTH_OK 2024-04-15T22:20:01.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:00 smithi176 ceph-mon[25776]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-15T22:20:01.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:00 smithi176 ceph-mon[25776]: osdmap e11: 2 total, 1 up, 2 in 2024-04-15T22:20:01.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:00 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:20:01.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:00 smithi176 ceph-mon[25776]: from='osd.1 [v2:172.21.15.92:6810/1540392933,v1:172.21.15.92:6811/1540392933]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:01.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:00 smithi176 ceph-mon[25776]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:01.464 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:20:01 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1[35886]: 2024-04-15T22:20:01.107+0000 7f9fa3ae8700 -1 osd.1 0 waiting for initial osdmap 2024-04-15T22:20:01.465 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:20:01 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1[35886]: 2024-04-15T22:20:01.112+0000 7f9f9c47b700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:20:01.578 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-15T22:20:01.578 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-15T22:20:01.578 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-04-15T22:20:01.578 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-04-15T22:20:01.578 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128543 s, 816 MB/s 2024-04-15T22:20:01.578 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-04-15T22:20:02.211 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:02 smithi092 ceph-mon[24813]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-15T22:20:02.211 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:02 smithi092 ceph-mon[24813]: osdmap e12: 2 total, 1 up, 2 in 2024-04-15T22:20:02.211 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:02 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:20:02.212 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:02 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:20:02.212 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:02 smithi092 ceph-mon[24813]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:20:02.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:02 smithi092 ceph-mon[29727]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-15T22:20:02.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:02 smithi092 ceph-mon[29727]: osdmap e12: 2 total, 1 up, 2 in 2024-04-15T22:20:02.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:02 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:20:02.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:02 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:20:02.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:02 smithi092 ceph-mon[29727]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:20:02.237 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi092:vg_nvme/lv_2 2024-04-15T22:20:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:02 smithi176 ceph-mon[25776]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-15T22:20:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:02 smithi176 ceph-mon[25776]: osdmap e12: 2 total, 1 up, 2 in 2024-04-15T22:20:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:02 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:20:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:02 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:20:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:02 smithi176 ceph-mon[25776]: pgmap v54: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T22:20:03.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:03 smithi092 ceph-mon[24813]: purged_snaps scrub starts 2024-04-15T22:20:03.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:03 smithi092 ceph-mon[24813]: purged_snaps scrub ok 2024-04-15T22:20:03.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:03 smithi092 ceph-mon[24813]: osd.1 [v2:172.21.15.92:6810/1540392933,v1:172.21.15.92:6811/1540392933] boot 2024-04-15T22:20:03.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:03 smithi092 ceph-mon[24813]: osdmap e13: 2 total, 2 up, 2 in 2024-04-15T22:20:03.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:20:03.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:03 smithi092 ceph-mon[29727]: purged_snaps scrub starts 2024-04-15T22:20:03.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:03 smithi092 ceph-mon[29727]: purged_snaps scrub ok 2024-04-15T22:20:03.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:03 smithi092 ceph-mon[29727]: osd.1 [v2:172.21.15.92:6810/1540392933,v1:172.21.15.92:6811/1540392933] boot 2024-04-15T22:20:03.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:03 smithi092 ceph-mon[29727]: osdmap e13: 2 total, 2 up, 2 in 2024-04-15T22:20:03.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:20:03.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:03 smithi176 ceph-mon[25776]: purged_snaps scrub starts 2024-04-15T22:20:03.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:03 smithi176 ceph-mon[25776]: purged_snaps scrub ok 2024-04-15T22:20:03.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:03 smithi176 ceph-mon[25776]: osd.1 [v2:172.21.15.92:6810/1540392933,v1:172.21.15.92:6811/1540392933] boot 2024-04-15T22:20:03.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:03 smithi176 ceph-mon[25776]: osdmap e13: 2 total, 2 up, 2 in 2024-04-15T22:20:03.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:03 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:20:04.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:04 smithi092 ceph-mon[24813]: osdmap e14: 2 total, 2 up, 2 in 2024-04-15T22:20:04.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:04 smithi092 ceph-mon[24813]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:04.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:04 smithi092 ceph-mon[29727]: osdmap e14: 2 total, 2 up, 2 in 2024-04-15T22:20:04.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:04 smithi092 ceph-mon[29727]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:04.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:04 smithi176 ceph-mon[25776]: osdmap e14: 2 total, 2 up, 2 in 2024-04-15T22:20:04.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:04 smithi176 ceph-mon[25776]: pgmap v57: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:04.811 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:20:04 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:20:04] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:20:05.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[24813]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:20:05.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:20:05.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:20:05.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:05.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:05.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[24813]: Detected new or changed devices on smithi092 2024-04-15T22:20:05.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:05.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:05.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:05.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:05.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[24813]: pgmap v58: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:05.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:05.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[29727]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:20:05.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:20:05.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:20:05.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:05.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:05.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[29727]: Detected new or changed devices on smithi092 2024-04-15T22:20:05.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:05.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:05.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:05.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:05.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[29727]: pgmap v58: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:05.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:05 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:05.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:05 smithi176 ceph-mon[25776]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:20:05.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:05 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:20:05.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:05 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:20:05.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:05 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:05.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:05 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:05.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:05 smithi176 ceph-mon[25776]: Detected new or changed devices on smithi092 2024-04-15T22:20:05.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:05 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:05.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:05 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:05.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:05 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:05.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:05 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:05.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:05 smithi176 ceph-mon[25776]: pgmap v58: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:05.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:05 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:06.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:06 smithi092 ceph-mon[24813]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:20:06.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:06 smithi092 ceph-mon[29727]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:20:06.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:06 smithi176 ceph-mon[25776]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:20:07.212 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:07 smithi092 ceph-mon[24813]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:07.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:07 smithi092 ceph-mon[29727]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:07.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:07 smithi176 ceph-mon[25776]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:07.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:20:07 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:20:07] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:20:08.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:08.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:08.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:08.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:08.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:08.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[24813]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:08.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:08.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3109912143' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c726316c-7073-4fc1-b4b1-07dd1f3acb3b"}]: dispatch 2024-04-15T22:20:08.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3109912143' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c726316c-7073-4fc1-b4b1-07dd1f3acb3b"}]': finished 2024-04-15T22:20:08.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[24813]: osdmap e15: 3 total, 2 up, 3 in 2024-04-15T22:20:08.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:08.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2422753886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:20:08.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:08.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:08.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:08.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:08.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:08.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[29727]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:08.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:08.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3109912143' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c726316c-7073-4fc1-b4b1-07dd1f3acb3b"}]: dispatch 2024-04-15T22:20:08.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3109912143' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c726316c-7073-4fc1-b4b1-07dd1f3acb3b"}]': finished 2024-04-15T22:20:08.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[29727]: osdmap e15: 3 total, 2 up, 3 in 2024-04-15T22:20:08.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:08.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:08 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2422753886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:20:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:08 smithi176 ceph-mon[25776]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:08 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3109912143' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c726316c-7073-4fc1-b4b1-07dd1f3acb3b"}]: dispatch 2024-04-15T22:20:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:08 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3109912143' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c726316c-7073-4fc1-b4b1-07dd1f3acb3b"}]': finished 2024-04-15T22:20:08.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:08 smithi176 ceph-mon[25776]: osdmap e15: 3 total, 2 up, 3 in 2024-04-15T22:20:08.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:08.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:08 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2422753886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:20:09.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:09 smithi092 ceph-mon[29727]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:20:09.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:09 smithi092 ceph-mon[29727]: Cluster is now healthy 2024-04-15T22:20:09.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:09 smithi092 ceph-mon[24813]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:20:09.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:09 smithi092 ceph-mon[24813]: Cluster is now healthy 2024-04-15T22:20:09.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:09 smithi176 ceph-mon[25776]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:20:09.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:09 smithi176 ceph-mon[25776]: Cluster is now healthy 2024-04-15T22:20:10.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:10 smithi092 ceph-mon[29727]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:10.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:10 smithi092 ceph-mon[24813]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:10.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:10 smithi176 ceph-mon[25776]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:12.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:12 smithi092 ceph-mon[24813]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:12.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:12 smithi092 ceph-mon[29727]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:12.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:12 smithi176 ceph-mon[25776]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:14.953 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:14 smithi092 ceph-mon[24813]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:14.954 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:14 smithi092 ceph-mon[29727]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:14.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:14 smithi176 ceph-mon[25776]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:15.663 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:15 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-15T22:20:15.663 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:15 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:15.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:15 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-15T22:20:15.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:15 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:15.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:15 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-15T22:20:15.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:15 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:16.931 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:16 smithi092 ceph-mon[24813]: Deploying daemon osd.2 on smithi092 2024-04-15T22:20:16.931 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:16 smithi092 ceph-mon[24813]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:16.932 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:16 smithi092 ceph-mon[29727]: Deploying daemon osd.2 on smithi092 2024-04-15T22:20:16.932 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:16 smithi092 ceph-mon[29727]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:16.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:16 smithi176 ceph-mon[25776]: Deploying daemon osd.2 on smithi092 2024-04-15T22:20:16.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:16 smithi176 ceph-mon[25776]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:17.590 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:20:17 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:20:17] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:20:18.460 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:18.461 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:18.461 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:18 smithi092 ceph-mon[24813]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:18.461 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:20:18.461 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:20:18.461 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:18.461 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:18.461 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:18 smithi092 ceph-mon[29727]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:18.461 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:20:18.461 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:20:18.482 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:18.482 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:18.482 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:18 smithi176 ceph-mon[25776]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:18.482 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:20:18.482 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:20:20.200 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:20 smithi092 ceph-mon[24813]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:20.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:20 smithi092 ceph-mon[29727]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:20.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:20 smithi176 ceph-mon[25776]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:21.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:21 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:21.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:21 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:21.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:21 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:21.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:21 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:21.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:21 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:21.624 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:21 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:21.624 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:21 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:21.624 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:21 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:21.624 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:21 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:21.624 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:21 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:21.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:21 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:21.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:21 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:21.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:21 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:21.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:21 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:21.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:21 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:22.637 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:22 smithi092 ceph-mon[24813]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:22.637 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:22 smithi092 ceph-mon[29727]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:22.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:22 smithi176 ceph-mon[25776]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:22.971 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 2 on host 'smithi092' 2024-04-15T22:20:23.597 DEBUG:teuthology.orchestra.run.smithi092:osd.2> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.2.service 2024-04-15T22:20:23.601 INFO:tasks.cephadm:Deploying osd.3 on smithi092 with /dev/vg_nvme/lv_1... 2024-04-15T22:20:23.601 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-15T22:20:24.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:24.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:24.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:24.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:24.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:24.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:24.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:24.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[29727]: Adjusting osd_memory_target on smithi092 to 5013M 2024-04-15T22:20:24.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:24.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:24.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[29727]: from='osd.2 [v2:172.21.15.92:6818/2065441487,v1:172.21.15.92:6819/2065441487]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T22:20:24.200 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[29727]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T22:20:24.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[29727]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:24.201 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:24.201 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:24.201 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:24.201 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:24.201 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:24.201 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:24.201 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:24.201 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[24813]: Adjusting osd_memory_target on smithi092 to 5013M 2024-04-15T22:20:24.201 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:24.201 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:24.201 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[24813]: from='osd.2 [v2:172.21.15.92:6818/2065441487,v1:172.21.15.92:6819/2065441487]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T22:20:24.201 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[24813]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T22:20:24.202 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:23 smithi092 ceph-mon[24813]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:24.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:23 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:24.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:23 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:24.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:23 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:24.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:23 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:24.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:23 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:24.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:23 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:24.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:23 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:24.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:23 smithi176 ceph-mon[25776]: Adjusting osd_memory_target on smithi092 to 5013M 2024-04-15T22:20:24.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:23 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:24.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:23 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:24.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:23 smithi176 ceph-mon[25776]: from='osd.2 [v2:172.21.15.92:6818/2065441487,v1:172.21.15.92:6819/2065441487]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T22:20:24.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:23 smithi176 ceph-mon[25776]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T22:20:24.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:23 smithi176 ceph-mon[25776]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:25.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:25 smithi092 ceph-mon[24813]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T22:20:25.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:25 smithi092 ceph-mon[24813]: osdmap e16: 3 total, 2 up, 3 in 2024-04-15T22:20:25.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:25 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:25.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:25 smithi092 ceph-mon[24813]: from='osd.2 [v2:172.21.15.92:6818/2065441487,v1:172.21.15.92:6819/2065441487]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:25.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:25 smithi092 ceph-mon[24813]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:25.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:25 smithi092 ceph-mon[29727]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T22:20:25.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:25 smithi092 ceph-mon[29727]: osdmap e16: 3 total, 2 up, 3 in 2024-04-15T22:20:25.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:25 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:25.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:25 smithi092 ceph-mon[29727]: from='osd.2 [v2:172.21.15.92:6818/2065441487,v1:172.21.15.92:6819/2065441487]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:25.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:25 smithi092 ceph-mon[29727]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:25.216 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:20:25 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2[38836]: 2024-04-15T22:20:25.012+0000 7f8ced9d6700 -1 osd.2 0 waiting for initial osdmap 2024-04-15T22:20:25.216 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:20:25 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2[38836]: 2024-04-15T22:20:25.015+0000 7f8cea371700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:20:25.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:25 smithi176 ceph-mon[25776]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T22:20:25.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:25 smithi176 ceph-mon[25776]: osdmap e16: 3 total, 2 up, 3 in 2024-04-15T22:20:25.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:25 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:25.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:25 smithi176 ceph-mon[25776]: from='osd.2 [v2:172.21.15.92:6818/2065441487,v1:172.21.15.92:6819/2065441487]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:25.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:25 smithi176 ceph-mon[25776]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:25.632 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-15T22:20:25.632 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-15T22:20:25.632 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-04-15T22:20:25.632 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-04-15T22:20:25.632 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0131211 s, 799 MB/s 2024-04-15T22:20:25.633 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 2024-04-15T22:20:25.633 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-04-15T22:20:26.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:26 smithi092 ceph-mon[29727]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-15T22:20:26.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:26 smithi092 ceph-mon[29727]: osdmap e17: 3 total, 2 up, 3 in 2024-04-15T22:20:26.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:26 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:26.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:26 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:26.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:26 smithi092 ceph-mon[29727]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:26.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:26 smithi092 ceph-mon[24813]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-15T22:20:26.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:26 smithi092 ceph-mon[24813]: osdmap e17: 3 total, 2 up, 3 in 2024-04-15T22:20:26.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:26 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:26.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:26 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:26.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:26 smithi092 ceph-mon[24813]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:26.248 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi092:vg_nvme/lv_1 2024-04-15T22:20:26.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:26 smithi176 ceph-mon[25776]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-15T22:20:26.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:26 smithi176 ceph-mon[25776]: osdmap e17: 3 total, 2 up, 3 in 2024-04-15T22:20:26.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:26 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:26.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:26 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:26.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:26 smithi176 ceph-mon[25776]: pgmap v72: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T22:20:27.192 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:27 smithi092 ceph-mon[29727]: purged_snaps scrub starts 2024-04-15T22:20:27.192 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:27 smithi092 ceph-mon[29727]: purged_snaps scrub ok 2024-04-15T22:20:27.192 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:27 smithi092 ceph-mon[29727]: osd.2 [v2:172.21.15.92:6818/2065441487,v1:172.21.15.92:6819/2065441487] boot 2024-04-15T22:20:27.192 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:27 smithi092 ceph-mon[29727]: osdmap e18: 3 total, 3 up, 3 in 2024-04-15T22:20:27.193 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:27 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:27.193 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:27 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:27.193 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:27 smithi092 ceph-mon[24813]: purged_snaps scrub starts 2024-04-15T22:20:27.193 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:27 smithi092 ceph-mon[24813]: purged_snaps scrub ok 2024-04-15T22:20:27.193 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:27 smithi092 ceph-mon[24813]: osd.2 [v2:172.21.15.92:6818/2065441487,v1:172.21.15.92:6819/2065441487] boot 2024-04-15T22:20:27.193 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:27 smithi092 ceph-mon[24813]: osdmap e18: 3 total, 3 up, 3 in 2024-04-15T22:20:27.193 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:27.193 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:27.380 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:27 smithi176 ceph-mon[25776]: purged_snaps scrub starts 2024-04-15T22:20:27.380 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:27 smithi176 ceph-mon[25776]: purged_snaps scrub ok 2024-04-15T22:20:27.380 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:27 smithi176 ceph-mon[25776]: osd.2 [v2:172.21.15.92:6818/2065441487,v1:172.21.15.92:6819/2065441487] boot 2024-04-15T22:20:27.380 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:27 smithi176 ceph-mon[25776]: osdmap e18: 3 total, 3 up, 3 in 2024-04-15T22:20:27.380 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:27 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:27.380 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:27 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:20:27.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:20:27 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:20:27] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:20:28.257 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[24813]: osdmap e19: 3 total, 3 up, 3 in 2024-04-15T22:20:28.257 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[24813]: pgmap v75: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:28.257 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:28.257 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[24813]: Detected new or changed devices on smithi092 2024-04-15T22:20:28.257 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:28.258 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-15T22:20:28.258 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:28.258 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:28.258 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:28.258 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:28.258 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[29727]: osdmap e19: 3 total, 3 up, 3 in 2024-04-15T22:20:28.258 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[29727]: pgmap v75: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:28.258 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:28.258 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[29727]: Detected new or changed devices on smithi092 2024-04-15T22:20:28.258 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:28.258 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-15T22:20:28.258 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:28.259 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:28.259 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:28.259 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:28.392 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:28 smithi176 ceph-mon[25776]: osdmap e19: 3 total, 3 up, 3 in 2024-04-15T22:20:28.392 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:28 smithi176 ceph-mon[25776]: pgmap v75: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:28.392 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:28.392 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:28 smithi176 ceph-mon[25776]: Detected new or changed devices on smithi092 2024-04-15T22:20:28.392 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:28.393 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-15T22:20:28.393 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:28.393 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:28.393 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:28.393 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:29.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:29 smithi176 ceph-mon[25776]: pgmap v76: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:29.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:29 smithi176 ceph-mon[25776]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:20:29.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:29 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-15T22:20:29.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:29 smithi176 ceph-mon[25776]: osdmap e20: 3 total, 3 up, 3 in 2024-04-15T22:20:29.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:29 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T22:20:29.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:29 smithi176 ceph-mon[25776]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:20:29.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:29 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:20:29.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:29 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:20:29.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:29 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:29.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[29727]: pgmap v76: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:29.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[29727]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:20:29.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-15T22:20:29.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[29727]: osdmap e20: 3 total, 3 up, 3 in 2024-04-15T22:20:29.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T22:20:29.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[29727]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:20:29.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:20:29.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:20:29.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:29.325 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[24813]: pgmap v76: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:29.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[24813]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:20:29.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-15T22:20:29.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[24813]: osdmap e20: 3 total, 3 up, 3 in 2024-04-15T22:20:29.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T22:20:29.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[24813]: from='client.14295 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:20:29.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:20:29.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:20:29.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:29 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:30.350 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:20:30 smithi092 sudo[40169]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:20:30.350 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:20:30 smithi092 sudo[40169]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:20:30.350 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:20:30 smithi092 sudo[40169]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:20:30.635 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:20:30 smithi092 sudo[40191]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:20:30.635 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:20:30 smithi092 sudo[40191]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:20:30.635 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:20:30 smithi092 sudo[40191]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:20:30.900 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:30 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-15T22:20:30.900 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:30 smithi092 ceph-mon[24813]: osdmap e21: 3 total, 3 up, 3 in 2024-04-15T22:20:30.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:30 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:30.901 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:30 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-15T22:20:30.901 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:30 smithi092 ceph-mon[29727]: osdmap e21: 3 total, 3 up, 3 in 2024-04-15T22:20:30.901 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:30 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:30.901 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:20:30 smithi092 sudo[40203]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:20:30.901 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:20:30 smithi092 sudo[40203]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:20:30.901 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:20:30 smithi092 sudo[40203]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:20:30.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:30 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-15T22:20:30.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:30 smithi176 ceph-mon[25776]: osdmap e21: 3 total, 3 up, 3 in 2024-04-15T22:20:30.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:30 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:31.161 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:20:30 smithi092 sudo[40224]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:20:31.162 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:20:30 smithi092 sudo[40224]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:20:31.162 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:20:30 smithi092 sudo[40224]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:20:31.424 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:20:31] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:20:31.424 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:20:31 smithi092 sudo[40292]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:20:31.424 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:20:31 smithi092 sudo[40292]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:20:31.425 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:20:31 smithi092 sudo[40292]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:20:31.684 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 sudo[40322]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:20:31.684 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 sudo[40322]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:20:31.684 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:20:31 smithi092 sudo[40319]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:20:31.684 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:20:31 smithi092 sudo[40319]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:20:31.684 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:20:31 smithi092 sudo[40319]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:20:31.936 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 sudo[40322]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:20:31.936 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 sudo[40344]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:20:31.936 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 sudo[40344]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:20:31.936 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 sudo[40344]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:20:32.188 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[24813]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:32.188 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[24813]: osdmap e22: 3 total, 3 up, 3 in 2024-04-15T22:20:32.188 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:32.188 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:32.189 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:32.189 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:32.189 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:32.189 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:32.189 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[24813]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:20:32.189 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[29727]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:32.189 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[29727]: osdmap e22: 3 total, 3 up, 3 in 2024-04-15T22:20:32.189 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:32.189 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:32.189 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:32.189 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:32.189 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:32.189 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:32.190 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:31 smithi092 ceph-mon[29727]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:20:32.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:31 smithi176 ceph-mon[25776]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:32.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:31 smithi176 ceph-mon[25776]: osdmap e22: 3 total, 3 up, 3 in 2024-04-15T22:20:32.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:31 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:32.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:31 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:32.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:31 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:32.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:31 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:32.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:31 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:32.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:31 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:32.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:31 smithi176 ceph-mon[25776]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:20:32.453 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 sudo[40354]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:20:32.453 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 sudo[40354]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:20:32.454 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 sudo[40354]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:20:32.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 sudo[40366]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:20:32.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 sudo[40366]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:20:32.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 sudo[40366]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:20:32.952 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 sudo[28205]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T22:20:32.953 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 sudo[28205]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:20:32.953 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 sudo[28205]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:20:33.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 sudo[28208]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T22:20:33.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 sudo[28208]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T22:20:33.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 sudo[28208]: pam_unix(sudo:session): session closed for user root 2024-04-15T22:20:33.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:33.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:20:33.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:20:33.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:20:33.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:20:33.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:20:33.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:20:33.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:20:33.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:20:33.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:20:33.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: Cluster is now healthy 2024-04-15T22:20:33.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:20:33.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:20:33.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:20:33.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:20:33.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:32 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/4231679844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6d08992-1892-4188-b2d5-39ca9813b173"}]: dispatch 2024-04-15T22:20:33.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:33.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:20:33.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:20:33.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:20:33.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:20:33.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:20:33.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:20:33.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:20:33.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:20:33.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:20:33.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: Cluster is now healthy 2024-04-15T22:20:33.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:20:33.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:20:33.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:20:33.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:20:33.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/4231679844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6d08992-1892-4188-b2d5-39ca9813b173"}]: dispatch 2024-04-15T22:20:33.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:33.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:20:33.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:20:33.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:20:33.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:20:33.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:20:33.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:20:33.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:20:33.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:20:33.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:20:33.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: Cluster is now healthy 2024-04-15T22:20:33.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:20:33.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:20:33.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:20:33.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:20:33.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:32 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/4231679844' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6d08992-1892-4188-b2d5-39ca9813b173"}]: dispatch 2024-04-15T22:20:34.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:33 smithi176 ceph-mon[25776]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:20:34.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:33 smithi176 ceph-mon[25776]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:20:34.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:33 smithi176 ceph-mon[25776]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-15T22:20:34.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:33 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/4231679844' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6d08992-1892-4188-b2d5-39ca9813b173"}]': finished 2024-04-15T22:20:34.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:33 smithi176 ceph-mon[25776]: osdmap e23: 4 total, 3 up, 4 in 2024-04-15T22:20:34.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:33 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:34.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:33 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:34.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:33 smithi176 ceph-mon[25776]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:34.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:33 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2658060572' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:20:34.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[24813]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:20:34.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[24813]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:20:34.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[24813]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-15T22:20:34.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/4231679844' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6d08992-1892-4188-b2d5-39ca9813b173"}]': finished 2024-04-15T22:20:34.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[24813]: osdmap e23: 4 total, 3 up, 4 in 2024-04-15T22:20:34.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:34.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:34.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[24813]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:34.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2658060572' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:20:34.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[29727]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T22:20:34.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[29727]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T22:20:34.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[29727]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-15T22:20:34.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/4231679844' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6d08992-1892-4188-b2d5-39ca9813b173"}]': finished 2024-04-15T22:20:34.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[29727]: osdmap e23: 4 total, 3 up, 4 in 2024-04-15T22:20:34.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:34.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:34.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[29727]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:34.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:33 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2658060572' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:20:36.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:36 smithi092 ceph-mon[24813]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:36.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:36 smithi092 ceph-mon[29727]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:36.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:36 smithi176 ceph-mon[25776]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:37.581 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:20:37 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:20:37] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:20:38.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:38 smithi092 ceph-mon[24813]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 184 KiB/s wr, 13 op/s 2024-04-15T22:20:38.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:38.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:38.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:38 smithi092 ceph-mon[29727]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 184 KiB/s wr, 13 op/s 2024-04-15T22:20:38.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:38 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:38.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:38 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:38.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:38 smithi176 ceph-mon[25776]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 184 KiB/s wr, 13 op/s 2024-04-15T22:20:38.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:38 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:38.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:38 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:40.327 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:40 smithi176 ceph-mon[25776]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.9 KiB/s rd, 125 KiB/s wr, 8 op/s 2024-04-15T22:20:40.327 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:40 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-15T22:20:40.327 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:40 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:40.327 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:40 smithi176 ceph-mon[25776]: Deploying daemon osd.3 on smithi092 2024-04-15T22:20:40.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:40 smithi092 ceph-mon[24813]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.9 KiB/s rd, 125 KiB/s wr, 8 op/s 2024-04-15T22:20:40.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:40 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-15T22:20:40.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:40 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:40.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:40 smithi092 ceph-mon[24813]: Deploying daemon osd.3 on smithi092 2024-04-15T22:20:40.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:40 smithi092 ceph-mon[29727]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.9 KiB/s rd, 125 KiB/s wr, 8 op/s 2024-04-15T22:20:40.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:40 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-15T22:20:40.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:40 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:40.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:40 smithi092 ceph-mon[29727]: Deploying daemon osd.3 on smithi092 2024-04-15T22:20:42.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:42 smithi092 ceph-mon[29727]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 95 KiB/s wr, 6 op/s 2024-04-15T22:20:42.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:42 smithi092 ceph-mon[24813]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 95 KiB/s wr, 6 op/s 2024-04-15T22:20:42.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:42 smithi176 ceph-mon[25776]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 95 KiB/s wr, 6 op/s 2024-04-15T22:20:44.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:44 smithi092 ceph-mon[24813]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-04-15T22:20:44.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:44 smithi092 ceph-mon[29727]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-04-15T22:20:44.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:44 smithi176 ceph-mon[25776]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-04-15T22:20:45.633 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:45 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:45.633 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:45 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:45.633 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:45 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:45.633 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:45 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:45.633 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:45 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:45.633 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:45 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:45.633 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:45 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:45.633 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:45 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:45.633 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:45 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:45.634 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:45 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:45.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:45 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:45.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:45 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:45.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:45 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:45.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:45 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:45.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:45 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:46.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:46 smithi092 ceph-mon[24813]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-15T22:20:46.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:46 smithi092 ceph-mon[29727]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-15T22:20:46.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:46 smithi176 ceph-mon[25776]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-15T22:20:47.089 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 3 on host 'smithi092' 2024-04-15T22:20:48.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:48.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:48.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:48.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:48.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:48.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:48.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:48.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:48.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[29727]: Adjusting osd_memory_target on smithi092 to 3760M 2024-04-15T22:20:48.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:48.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:48.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[29727]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-15T22:20:48.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[29727]: from='osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T22:20:48.339 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:48.339 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:48.339 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:48.339 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:48.340 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:48.340 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:48.340 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:48.340 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:48.340 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[24813]: Adjusting osd_memory_target on smithi092 to 3760M 2024-04-15T22:20:48.340 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:48.340 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:48.340 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[24813]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-15T22:20:48.340 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:48 smithi092 ceph-mon[24813]: from='osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T22:20:48.365 DEBUG:teuthology.orchestra.run.smithi092:osd.3> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.3.service 2024-04-15T22:20:48.367 INFO:tasks.cephadm:Deploying osd.4 on smithi176 with /dev/vg_nvme/lv_4... 2024-04-15T22:20:48.368 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-15T22:20:48.402 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:48 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:48.402 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:48 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:48.402 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:48 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:48.402 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:48 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:48.402 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:48 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:48.402 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:48 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:48.402 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:48 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:48.402 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:48 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:20:48.402 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:48 smithi176 ceph-mon[25776]: Adjusting osd_memory_target on smithi092 to 3760M 2024-04-15T22:20:48.402 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:48 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:48.402 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:48 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:48.402 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:48 smithi176 ceph-mon[25776]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-15T22:20:48.402 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:48 smithi176 ceph-mon[25776]: from='osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T22:20:49.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:49 smithi092 ceph-mon[24813]: from='osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T22:20:49.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:49 smithi092 ceph-mon[24813]: osdmap e24: 4 total, 3 up, 4 in 2024-04-15T22:20:49.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:49 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:49.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:49 smithi092 ceph-mon[24813]: from='osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:49.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:49 smithi092 ceph-mon[29727]: from='osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T22:20:49.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:49 smithi092 ceph-mon[29727]: osdmap e24: 4 total, 3 up, 4 in 2024-04-15T22:20:49.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:49 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:49.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:49 smithi092 ceph-mon[29727]: from='osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:49.466 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:20:49 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3[41854]: 2024-04-15T22:20:49.139+0000 7f7fe8296700 -1 osd.3 0 waiting for initial osdmap 2024-04-15T22:20:49.466 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:20:49 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3[41854]: 2024-04-15T22:20:49.144+0000 7f7fe0c29700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:20:49.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:49 smithi176 ceph-mon[25776]: from='osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T22:20:49.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:49 smithi176 ceph-mon[25776]: osdmap e24: 4 total, 3 up, 4 in 2024-04-15T22:20:49.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:49 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:49.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:49 smithi176 ceph-mon[25776]: from='osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-04-15T22:20:50.380 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-15T22:20:50.380 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-15T22:20:50.380 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-04-15T22:20:50.380 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-04-15T22:20:50.380 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115728 s, 906 MB/s 2024-04-15T22:20:50.381 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-04-15T22:20:50.381 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:50 smithi176 ceph-mon[25776]: from='osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-15T22:20:50.381 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:50 smithi176 ceph-mon[25776]: osdmap e25: 4 total, 3 up, 4 in 2024-04-15T22:20:50.381 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:50 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:50.381 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:50 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:50.381 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:50 smithi176 ceph-mon[25776]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:50.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:50 smithi092 ceph-mon[29727]: from='osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-15T22:20:50.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:50 smithi092 ceph-mon[29727]: osdmap e25: 4 total, 3 up, 4 in 2024-04-15T22:20:50.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:50 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:50.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:50 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:50.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:50 smithi092 ceph-mon[29727]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:50.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:50 smithi092 ceph-mon[24813]: from='osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-04-15T22:20:50.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:50 smithi092 ceph-mon[24813]: osdmap e25: 4 total, 3 up, 4 in 2024-04-15T22:20:50.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:50 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:50.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:50 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:50.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:50 smithi092 ceph-mon[24813]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-15T22:20:51.002 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi176:vg_nvme/lv_4 2024-04-15T22:20:51.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:51 smithi176 ceph-mon[25776]: purged_snaps scrub starts 2024-04-15T22:20:51.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:51 smithi176 ceph-mon[25776]: purged_snaps scrub ok 2024-04-15T22:20:51.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:51 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:51.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:51 smithi176 ceph-mon[25776]: osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626] boot 2024-04-15T22:20:51.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:51 smithi176 ceph-mon[25776]: osdmap e26: 4 total, 4 up, 4 in 2024-04-15T22:20:51.353 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:51 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:51.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:20:51] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:20:51.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:20:51] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:20:51.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-mon[24813]: purged_snaps scrub starts 2024-04-15T22:20:51.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-mon[24813]: purged_snaps scrub ok 2024-04-15T22:20:51.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:51.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-mon[24813]: osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626] boot 2024-04-15T22:20:51.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-mon[24813]: osdmap e26: 4 total, 4 up, 4 in 2024-04-15T22:20:51.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:51.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-mon[29727]: purged_snaps scrub starts 2024-04-15T22:20:51.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-mon[29727]: purged_snaps scrub ok 2024-04-15T22:20:51.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:51.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-mon[29727]: osd.3 [v2:172.21.15.92:6826/3902358626,v1:172.21.15.92:6827/3902358626] boot 2024-04-15T22:20:51.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-mon[29727]: osdmap e26: 4 total, 4 up, 4 in 2024-04-15T22:20:51.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:51 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:20:52.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: osdmap e27: 4 total, 4 up, 4 in 2024-04-15T22:20:52.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: Detected new or changed devices on smithi092 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:52.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:52.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: osdmap e27: 4 total, 4 up, 4 in 2024-04-15T22:20:52.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:52.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: Detected new or changed devices on smithi092 2024-04-15T22:20:52.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:52.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:52.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:52.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:52.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:52.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:52.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:52.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:52.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: osdmap e27: 4 total, 4 up, 4 in 2024-04-15T22:20:52.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:52.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: Detected new or changed devices on smithi092 2024-04-15T22:20:52.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:52.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:52.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:52.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:52.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:52.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:20:52.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:52.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:20:52.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:52.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:53.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:53 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:20:53.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:53 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:20:53.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:53 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:53.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:53 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:20:53.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:53 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:20:53.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:53 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:53.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:53 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:20:53.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:53 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:20:53.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:53 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:20:54.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:54 smithi176 ceph-mon[25776]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:20:54.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:54 smithi176 ceph-mon[25776]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:54.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:54 smithi092 ceph-mon[29727]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:20:54.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:54 smithi092 ceph-mon[29727]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:54.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:54 smithi092 ceph-mon[24813]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:20:54.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:54 smithi092 ceph-mon[24813]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:55.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:55 smithi092 ceph-mon[29727]: from='client.? 172.21.15.176:0/2004750209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e0ab27-4f3a-489a-beef-67a3dad86e2f"}]: dispatch 2024-04-15T22:20:55.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:55 smithi092 ceph-mon[29727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e0ab27-4f3a-489a-beef-67a3dad86e2f"}]: dispatch 2024-04-15T22:20:55.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:55 smithi092 ceph-mon[29727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15e0ab27-4f3a-489a-beef-67a3dad86e2f"}]': finished 2024-04-15T22:20:55.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:55 smithi092 ceph-mon[29727]: osdmap e28: 5 total, 4 up, 5 in 2024-04-15T22:20:55.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:55 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:20:55.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:55 smithi092 ceph-mon[24813]: from='client.? 172.21.15.176:0/2004750209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e0ab27-4f3a-489a-beef-67a3dad86e2f"}]: dispatch 2024-04-15T22:20:55.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:55 smithi092 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e0ab27-4f3a-489a-beef-67a3dad86e2f"}]: dispatch 2024-04-15T22:20:55.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:55 smithi092 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15e0ab27-4f3a-489a-beef-67a3dad86e2f"}]': finished 2024-04-15T22:20:55.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:55 smithi092 ceph-mon[24813]: osdmap e28: 5 total, 4 up, 5 in 2024-04-15T22:20:55.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:55 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:20:55.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:55 smithi176 ceph-mon[25776]: from='client.? 172.21.15.176:0/2004750209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e0ab27-4f3a-489a-beef-67a3dad86e2f"}]: dispatch 2024-04-15T22:20:55.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:55 smithi176 ceph-mon[25776]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15e0ab27-4f3a-489a-beef-67a3dad86e2f"}]: dispatch 2024-04-15T22:20:55.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:55 smithi176 ceph-mon[25776]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15e0ab27-4f3a-489a-beef-67a3dad86e2f"}]': finished 2024-04-15T22:20:55.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:55 smithi176 ceph-mon[25776]: osdmap e28: 5 total, 4 up, 5 in 2024-04-15T22:20:55.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:55 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:20:56.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:56 smithi092 ceph-mon[29727]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:56.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:56 smithi092 ceph-mon[29727]: from='client.? 172.21.15.176:0/3605964675' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:20:56.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:56 smithi092 ceph-mon[24813]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:56.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:56 smithi092 ceph-mon[24813]: from='client.? 172.21.15.176:0/3605964675' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:20:56.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:56 smithi176 ceph-mon[25776]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:56.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:56 smithi176 ceph-mon[25776]: from='client.? 172.21.15.176:0/3605964675' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:20:57.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:20:57 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:20:57] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:20:58.602 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:58 smithi176 ceph-mon[25776]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:58.602 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:58 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:58.603 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:20:58 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:58.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:58 smithi092 ceph-mon[29727]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:58.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:58 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:58.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:20:58 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:58.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:58 smithi092 ceph-mon[24813]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:20:58.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:58 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:20:58.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:20:58 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:00.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:00 smithi176 ceph-mon[25776]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:00.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:00 smithi092 ceph-mon[24813]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:00.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:00 smithi092 ceph-mon[29727]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:02.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:02 smithi176 ceph-mon[25776]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:02.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:02 smithi092 ceph-mon[24813]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:02.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:02 smithi092 ceph-mon[29727]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:03.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:03 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-15T22:21:03.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:03 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:03.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-15T22:21:03.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:03 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:03.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-15T22:21:03.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:03 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:04.784 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:04 smithi176 ceph-mon[25776]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:04.784 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:04 smithi176 ceph-mon[25776]: Deploying daemon osd.4 on smithi176 2024-04-15T22:21:04.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:04 smithi092 ceph-mon[24813]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:04.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:04 smithi092 ceph-mon[24813]: Deploying daemon osd.4 on smithi176 2024-04-15T22:21:04.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:04 smithi092 ceph-mon[29727]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:04.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:04 smithi092 ceph-mon[29727]: Deploying daemon osd.4 on smithi176 2024-04-15T22:21:06.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:06 smithi092 ceph-mon[24813]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:06.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:06 smithi092 ceph-mon[29727]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:06.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:06 smithi176 ceph-mon[25776]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:07.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:21:07 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:21:07] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:21:08.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[24813]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:08.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:08.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:08.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:08.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:08.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:08.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:08.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:08.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[29727]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:08.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:08.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:08.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:08.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:08.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:08.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:08.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:08.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:08 smithi176 ceph-mon[25776]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:08.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:10.343 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:10 smithi176 ceph-mon[25776]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:10.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:10 smithi092 ceph-mon[24813]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:10.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:10 smithi092 ceph-mon[29727]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:11.553 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 4 on host 'smithi176' 2024-04-15T22:21:12.238 DEBUG:teuthology.orchestra.run.smithi176:osd.4> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.4.service 2024-04-15T22:21:12.241 INFO:tasks.cephadm:Deploying osd.5 on smithi176 with /dev/vg_nvme/lv_3... 2024-04-15T22:21:12.241 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-15T22:21:12.827 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:12 smithi176 ceph-mon[25776]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:12.828 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:12.828 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:12.828 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:12.828 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:12.828 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:12.828 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:12 smithi176 ceph-mon[25776]: Adjusting osd_memory_target on smithi176 to 16064M 2024-04-15T22:21:12.828 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:12.828 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:12 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:12.828 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:12 smithi176 ceph-mon[25776]: from='osd.4 [v2:172.21.15.176:6800/3350225179,v1:172.21.15.176:6801/3350225179]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:21:12.828 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:12 smithi176 ceph-mon[25776]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:21:12.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[24813]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:12.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:12.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:12.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:12.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:12.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:12.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[24813]: Adjusting osd_memory_target on smithi176 to 16064M 2024-04-15T22:21:12.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:12.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:12.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[24813]: from='osd.4 [v2:172.21.15.176:6800/3350225179,v1:172.21.15.176:6801/3350225179]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:21:12.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[24813]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:21:12.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[29727]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:12.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:12.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:12.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:12.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:12.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:12.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[29727]: Adjusting osd_memory_target on smithi176 to 16064M 2024-04-15T22:21:12.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:12.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:12.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[29727]: from='osd.4 [v2:172.21.15.176:6800/3350225179,v1:172.21.15.176:6801/3350225179]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:21:12.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:12 smithi092 ceph-mon[29727]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T22:21:13.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:13 smithi176 ceph-mon[25776]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-15T22:21:13.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:13 smithi176 ceph-mon[25776]: osdmap e29: 5 total, 4 up, 5 in 2024-04-15T22:21:13.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:13 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:21:13.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:13 smithi176 ceph-mon[25776]: from='osd.4 [v2:172.21.15.176:6800/3350225179,v1:172.21.15.176:6801/3350225179]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:21:13.849 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:13 smithi176 ceph-mon[25776]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:21:13.849 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:21:13 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:21:13.594+0000 7f37ca349700 -1 osd.4 0 waiting for initial osdmap 2024-04-15T22:21:13.850 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:21:13 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:21:13.599+0000 7f37c2cdc700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:21:13.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:13 smithi092 ceph-mon[24813]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-15T22:21:13.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:13 smithi092 ceph-mon[24813]: osdmap e29: 5 total, 4 up, 5 in 2024-04-15T22:21:13.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:13 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:21:13.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:13 smithi092 ceph-mon[24813]: from='osd.4 [v2:172.21.15.176:6800/3350225179,v1:172.21.15.176:6801/3350225179]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:21:13.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:13 smithi092 ceph-mon[24813]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:21:13.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:13 smithi092 ceph-mon[29727]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-15T22:21:13.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:13 smithi092 ceph-mon[29727]: osdmap e29: 5 total, 4 up, 5 in 2024-04-15T22:21:13.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:13 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:21:13.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:13 smithi092 ceph-mon[29727]: from='osd.4 [v2:172.21.15.176:6800/3350225179,v1:172.21.15.176:6801/3350225179]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:21:13.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:13 smithi092 ceph-mon[29727]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:21:14.338 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-15T22:21:14.338 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-15T22:21:14.338 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-04-15T22:21:14.338 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-04-15T22:21:14.338 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128018 s, 819 MB/s 2024-04-15T22:21:14.338 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-04-15T22:21:14.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[29727]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:14.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[29727]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-04-15T22:21:14.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[29727]: osdmap e30: 5 total, 4 up, 5 in 2024-04-15T22:21:14.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:21:14.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:21:14.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[29727]: osd.4 [v2:172.21.15.176:6800/3350225179,v1:172.21.15.176:6801/3350225179] boot 2024-04-15T22:21:14.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[29727]: osdmap e31: 5 total, 5 up, 5 in 2024-04-15T22:21:14.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:21:14.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[24813]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:14.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[24813]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-04-15T22:21:14.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[24813]: osdmap e30: 5 total, 4 up, 5 in 2024-04-15T22:21:14.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:21:14.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:21:14.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[24813]: osd.4 [v2:172.21.15.176:6800/3350225179,v1:172.21.15.176:6801/3350225179] boot 2024-04-15T22:21:14.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[24813]: osdmap e31: 5 total, 5 up, 5 in 2024-04-15T22:21:14.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:14 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:21:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:14 smithi176 ceph-mon[25776]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T22:21:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:14 smithi176 ceph-mon[25776]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-04-15T22:21:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:14 smithi176 ceph-mon[25776]: osdmap e30: 5 total, 4 up, 5 in 2024-04-15T22:21:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:14 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:21:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:14 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:21:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:14 smithi176 ceph-mon[25776]: osd.4 [v2:172.21.15.176:6800/3350225179,v1:172.21.15.176:6801/3350225179] boot 2024-04-15T22:21:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:14 smithi176 ceph-mon[25776]: osdmap e31: 5 total, 5 up, 5 in 2024-04-15T22:21:14.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:14 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:21:15.004 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi176:vg_nvme/lv_3 2024-04-15T22:21:15.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:15 smithi092 ceph-mon[24813]: purged_snaps scrub starts 2024-04-15T22:21:15.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:15 smithi092 ceph-mon[24813]: purged_snaps scrub ok 2024-04-15T22:21:15.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:15 smithi092 ceph-mon[24813]: osdmap e32: 5 total, 5 up, 5 in 2024-04-15T22:21:15.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:15 smithi092 ceph-mon[29727]: purged_snaps scrub starts 2024-04-15T22:21:15.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:15 smithi092 ceph-mon[29727]: purged_snaps scrub ok 2024-04-15T22:21:15.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:15 smithi092 ceph-mon[29727]: osdmap e32: 5 total, 5 up, 5 in 2024-04-15T22:21:15.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:15 smithi176 ceph-mon[25776]: purged_snaps scrub starts 2024-04-15T22:21:15.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:15 smithi176 ceph-mon[25776]: purged_snaps scrub ok 2024-04-15T22:21:15.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:15 smithi176 ceph-mon[25776]: osdmap e32: 5 total, 5 up, 5 in 2024-04-15T22:21:16.896 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:16 smithi176 ceph-mon[25776]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:16.896 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:16 smithi176 ceph-mon[25776]: osdmap e33: 5 total, 5 up, 5 in 2024-04-15T22:21:16.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:16 smithi092 ceph-mon[24813]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:16.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:16 smithi092 ceph-mon[24813]: osdmap e33: 5 total, 5 up, 5 in 2024-04-15T22:21:16.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:16 smithi092 ceph-mon[29727]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:16.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:16 smithi092 ceph-mon[29727]: osdmap e33: 5 total, 5 up, 5 in 2024-04-15T22:21:17.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:17 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:21:17.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:17 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:21:17.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:17 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:17.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:17 smithi176 ceph-mon[25776]: osdmap e34: 5 total, 5 up, 5 in 2024-04-15T22:21:17.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:17 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:21:17.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:17 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:21:17.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:17 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:17.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:17 smithi092 ceph-mon[24813]: osdmap e34: 5 total, 5 up, 5 in 2024-04-15T22:21:17.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:21:17 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:21:17] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:21:17.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:21:17 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:21:17.659+0000 7f674e57c700 -1 calc_pg_upmaps abort due to max <= 0 2024-04-15T22:21:17.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:17 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:21:17.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:17 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:21:17.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:17 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:17.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:17 smithi092 ceph-mon[29727]: osdmap e34: 5 total, 5 up, 5 in 2024-04-15T22:21:18.613 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:21:18] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:21:18.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:21:18.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-04-15T22:21:18.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: Detected new or changed devices on smithi176 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: pgmap v119: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: osdmap e35: 5 total, 5 up, 5 in 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:18.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:21:18.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-04-15T22:21:18.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: Detected new or changed devices on smithi176 2024-04-15T22:21:18.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:18.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:18.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:18.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: pgmap v119: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-04-15T22:21:18.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:21:18.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:21:18.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:21:18.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: osdmap e35: 5 total, 5 up, 5 in 2024-04-15T22:21:18.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:18.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:18.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:18.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='client.24230 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:21:18.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: pgmap v118: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-04-15T22:21:18.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: Detected new or changed devices on smithi176 2024-04-15T22:21:18.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:18.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:18.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:18.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: pgmap v119: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 145 KiB/s, 0 objects/s recovering 2024-04-15T22:21:18.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:21:18.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:21:18.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:21:18.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: osdmap e35: 5 total, 5 up, 5 in 2024-04-15T22:21:18.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:18.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:18.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:18.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:18.980 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:19.863 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:19 smithi176 ceph-mon[25776]: pgmap v121: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 178 KiB/s, 0 objects/s recovering 2024-04-15T22:21:19.863 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:19 smithi176 ceph-mon[25776]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:21:19.863 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:19 smithi176 ceph-mon[25776]: Cluster is now healthy 2024-04-15T22:21:19.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:19 smithi092 ceph-mon[29727]: pgmap v121: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 178 KiB/s, 0 objects/s recovering 2024-04-15T22:21:19.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:19 smithi092 ceph-mon[29727]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:21:19.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:19 smithi092 ceph-mon[29727]: Cluster is now healthy 2024-04-15T22:21:19.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:19 smithi092 ceph-mon[24813]: pgmap v121: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 178 KiB/s, 0 objects/s recovering 2024-04-15T22:21:19.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:19 smithi092 ceph-mon[24813]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:21:19.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:19 smithi092 ceph-mon[24813]: Cluster is now healthy 2024-04-15T22:21:20.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:20 smithi092 ceph-mon[29727]: from='client.? 172.21.15.176:0/16287752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d6cda7f-0781-4201-95d2-1fc35a97d7ec"}]: dispatch 2024-04-15T22:21:20.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:20 smithi092 ceph-mon[29727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d6cda7f-0781-4201-95d2-1fc35a97d7ec"}]: dispatch 2024-04-15T22:21:20.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:20 smithi092 ceph-mon[29727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d6cda7f-0781-4201-95d2-1fc35a97d7ec"}]': finished 2024-04-15T22:21:20.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:20 smithi092 ceph-mon[29727]: osdmap e36: 6 total, 5 up, 6 in 2024-04-15T22:21:20.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:20 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:20.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:20 smithi092 ceph-mon[24813]: from='client.? 172.21.15.176:0/16287752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d6cda7f-0781-4201-95d2-1fc35a97d7ec"}]: dispatch 2024-04-15T22:21:20.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:20 smithi092 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d6cda7f-0781-4201-95d2-1fc35a97d7ec"}]: dispatch 2024-04-15T22:21:20.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:20 smithi092 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d6cda7f-0781-4201-95d2-1fc35a97d7ec"}]': finished 2024-04-15T22:21:20.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:20 smithi092 ceph-mon[24813]: osdmap e36: 6 total, 5 up, 6 in 2024-04-15T22:21:20.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:20 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:20.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:20 smithi176 ceph-mon[25776]: from='client.? 172.21.15.176:0/16287752' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d6cda7f-0781-4201-95d2-1fc35a97d7ec"}]: dispatch 2024-04-15T22:21:20.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:20 smithi176 ceph-mon[25776]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5d6cda7f-0781-4201-95d2-1fc35a97d7ec"}]: dispatch 2024-04-15T22:21:20.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:20 smithi176 ceph-mon[25776]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5d6cda7f-0781-4201-95d2-1fc35a97d7ec"}]': finished 2024-04-15T22:21:20.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:20 smithi176 ceph-mon[25776]: osdmap e36: 6 total, 5 up, 6 in 2024-04-15T22:21:20.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:20 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:21.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:21 smithi092 ceph-mon[29727]: pgmap v123: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:21.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:21 smithi092 ceph-mon[29727]: from='client.? 172.21.15.176:0/2379724685' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:21:21.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:21 smithi092 ceph-mon[24813]: pgmap v123: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:21.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:21 smithi092 ceph-mon[24813]: from='client.? 172.21.15.176:0/2379724685' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:21:21.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:21 smithi176 ceph-mon[25776]: pgmap v123: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:21.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:21 smithi176 ceph-mon[25776]: from='client.? 172.21.15.176:0/2379724685' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:21:23.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:23 smithi176 ceph-mon[25776]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:23.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:23 smithi092 ceph-mon[29727]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:23.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:23 smithi092 ceph-mon[24813]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:25.368 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:25 smithi176 ceph-mon[25776]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:25.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:25 smithi092 ceph-mon[24813]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:25.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:25 smithi092 ceph-mon[29727]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:27.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:27 smithi092 ceph-mon[24813]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:27.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-15T22:21:27.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:27.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:27 smithi092 ceph-mon[29727]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:27.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:27 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-15T22:21:27.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:27 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:27.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:27 smithi176 ceph-mon[25776]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:27.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:27 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-15T22:21:27.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:27 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:28.079 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:21:27 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:21:27] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:21:28.912 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:28 smithi176 ceph-mon[25776]: Deploying daemon osd.5 on smithi176 2024-04-15T22:21:28.912 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:28.912 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:29.055 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:28 smithi092 ceph-mon[24813]: Deploying daemon osd.5 on smithi176 2024-04-15T22:21:29.055 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:29.055 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:29.055 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:28 smithi092 ceph-mon[29727]: Deploying daemon osd.5 on smithi176 2024-04-15T22:21:29.055 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:29.055 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:29.963 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:29 smithi176 ceph-mon[25776]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:30.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:29 smithi092 ceph-mon[29727]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:30.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:29 smithi092 ceph-mon[24813]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:31.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:31 smithi176 ceph-mon[25776]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:32.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:31 smithi092 ceph-mon[24813]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:32.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:31 smithi092 ceph-mon[29727]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:33.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:33 smithi092 ceph-mon[24813]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:33.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:33 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:33.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:33 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:33.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:33 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:33.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:33 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:33.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:33 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:33.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:33 smithi092 ceph-mon[29727]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:33.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:33 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:33.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:33 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:33.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:33 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:33.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:33 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:33.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:33 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:33.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:33 smithi176 ceph-mon[25776]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:33.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:33 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:33.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:33 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:33.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:33 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:33.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:33 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:33.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:33 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:35.121 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 5 on host 'smithi176' 2024-04-15T22:21:35.134 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:35 smithi176 ceph-mon[25776]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:35.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:35 smithi092 ceph-mon[24813]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:35.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:35 smithi092 ceph-mon[29727]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:35.744 DEBUG:teuthology.orchestra.run.smithi176:osd.5> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.5.service 2024-04-15T22:21:35.747 INFO:tasks.cephadm:Deploying osd.6 on smithi176 with /dev/vg_nvme/lv_2... 2024-04-15T22:21:35.747 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-15T22:21:36.409 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:36.409 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:36.409 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:36.409 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:36.409 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:36.409 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:36.409 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:36 smithi176 ceph-mon[25776]: Adjusting osd_memory_target on smithi176 to 8032M 2024-04-15T22:21:36.409 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:36.409 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:36 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:36.409 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:36 smithi176 ceph-mon[25776]: from='osd.5 [v2:172.21.15.176:6808/2000788554,v1:172.21.15.176:6809/2000788554]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:21:36.410 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:36 smithi176 ceph-mon[25776]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:21:36.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:36.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:36.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:36.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:36.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:36.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:36.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[24813]: Adjusting osd_memory_target on smithi176 to 8032M 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[24813]: from='osd.5 [v2:172.21.15.176:6808/2000788554,v1:172.21.15.176:6809/2000788554]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[24813]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[29727]: Adjusting osd_memory_target on smithi176 to 8032M 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:36.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[29727]: from='osd.5 [v2:172.21.15.176:6808/2000788554,v1:172.21.15.176:6809/2000788554]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:21:36.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:36 smithi092 ceph-mon[29727]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T22:21:37.393 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:37 smithi176 ceph-mon[25776]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T22:21:37.393 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:37 smithi176 ceph-mon[25776]: osdmap e37: 6 total, 5 up, 6 in 2024-04-15T22:21:37.393 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:37 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:37.393 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:37 smithi176 ceph-mon[25776]: from='osd.5 [v2:172.21.15.176:6808/2000788554,v1:172.21.15.176:6809/2000788554]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:21:37.393 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:37 smithi176 ceph-mon[25776]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:21:37.393 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:37 smithi176 ceph-mon[25776]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:37.394 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:21:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:21:37.157+0000 7fdab4d75700 -1 osd.5 0 waiting for initial osdmap 2024-04-15T22:21:37.394 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:21:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:21:37.162+0000 7fdaadf09700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:21:37.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:37 smithi092 ceph-mon[29727]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T22:21:37.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:37 smithi092 ceph-mon[29727]: osdmap e37: 6 total, 5 up, 6 in 2024-04-15T22:21:37.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:37 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:37.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:37 smithi092 ceph-mon[29727]: from='osd.5 [v2:172.21.15.176:6808/2000788554,v1:172.21.15.176:6809/2000788554]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:21:37.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:37 smithi092 ceph-mon[29727]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:21:37.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:37 smithi092 ceph-mon[29727]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:37.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:37 smithi092 ceph-mon[24813]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T22:21:37.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:37 smithi092 ceph-mon[24813]: osdmap e37: 6 total, 5 up, 6 in 2024-04-15T22:21:37.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:37 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:37.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:37 smithi092 ceph-mon[24813]: from='osd.5 [v2:172.21.15.176:6808/2000788554,v1:172.21.15.176:6809/2000788554]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:21:37.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:37 smithi092 ceph-mon[24813]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:21:37.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:37 smithi092 ceph-mon[24813]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T22:21:37.886 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-15T22:21:37.887 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-15T22:21:37.887 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-04-15T22:21:37.887 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-04-15T22:21:37.887 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128032 s, 819 MB/s 2024-04-15T22:21:37.887 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-04-15T22:21:38.427 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[29727]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-04-15T22:21:38.427 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[29727]: osdmap e38: 6 total, 5 up, 6 in 2024-04-15T22:21:38.428 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:38.428 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:38.428 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[29727]: osd.5 [v2:172.21.15.176:6808/2000788554,v1:172.21.15.176:6809/2000788554] boot 2024-04-15T22:21:38.428 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[29727]: osdmap e39: 6 total, 6 up, 6 in 2024-04-15T22:21:38.428 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:38.428 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[24813]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-04-15T22:21:38.428 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[24813]: osdmap e38: 6 total, 5 up, 6 in 2024-04-15T22:21:38.428 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:38.428 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:38.428 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[24813]: osd.5 [v2:172.21.15.176:6808/2000788554,v1:172.21.15.176:6809/2000788554] boot 2024-04-15T22:21:38.428 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[24813]: osdmap e39: 6 total, 6 up, 6 in 2024-04-15T22:21:38.428 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:38.429 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:38 smithi176 ceph-mon[25776]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-04-15T22:21:38.429 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:38 smithi176 ceph-mon[25776]: osdmap e38: 6 total, 5 up, 6 in 2024-04-15T22:21:38.429 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:38 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:38.429 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:38 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:38.429 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:38 smithi176 ceph-mon[25776]: osd.5 [v2:172.21.15.176:6808/2000788554,v1:172.21.15.176:6809/2000788554] boot 2024-04-15T22:21:38.429 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:38 smithi176 ceph-mon[25776]: osdmap e39: 6 total, 6 up, 6 in 2024-04-15T22:21:38.429 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:38 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:21:38.532 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi176:vg_nvme/lv_2 2024-04-15T22:21:38.714 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:21:38 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:21:38] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:21:39.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[24813]: purged_snaps scrub starts 2024-04-15T22:21:39.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[24813]: purged_snaps scrub ok 2024-04-15T22:21:39.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[24813]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:39.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:39.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[24813]: Detected new or changed devices on smithi176 2024-04-15T22:21:39.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:39.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:39.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:39.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:39.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[24813]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:39.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:39.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[24813]: osdmap e40: 6 total, 6 up, 6 in 2024-04-15T22:21:39.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[24813]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:21:39.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[29727]: purged_snaps scrub starts 2024-04-15T22:21:39.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[29727]: purged_snaps scrub ok 2024-04-15T22:21:39.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[29727]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:39.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:39.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[29727]: Detected new or changed devices on smithi176 2024-04-15T22:21:39.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:39.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:39.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:39.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:39.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[29727]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:39.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:39.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[29727]: osdmap e40: 6 total, 6 up, 6 in 2024-04-15T22:21:39.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:39 smithi092 ceph-mon[29727]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:21:39.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:39 smithi176 ceph-mon[25776]: purged_snaps scrub starts 2024-04-15T22:21:39.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:39 smithi176 ceph-mon[25776]: purged_snaps scrub ok 2024-04-15T22:21:39.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:39 smithi176 ceph-mon[25776]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:39.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:39 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:39.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:39 smithi176 ceph-mon[25776]: Detected new or changed devices on smithi176 2024-04-15T22:21:39.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:39 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:39.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:39 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:39.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:39 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:39.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:39 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:39.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:39 smithi176 ceph-mon[25776]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:39.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:39 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:39.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:39 smithi176 ceph-mon[25776]: osdmap e40: 6 total, 6 up, 6 in 2024-04-15T22:21:39.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:39 smithi176 ceph-mon[25776]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:21:41.385 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:41 smithi176 ceph-mon[25776]: osdmap e41: 6 total, 6 up, 6 in 2024-04-15T22:21:41.386 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:41 smithi176 ceph-mon[25776]: pgmap v139: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:41.386 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:41 smithi176 ceph-mon[25776]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:21:41.386 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:41 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:21:41.386 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:41 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:21:41.386 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:41 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:41.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:21:41 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:21:41] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:21:41.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:41 smithi092 ceph-mon[29727]: osdmap e41: 6 total, 6 up, 6 in 2024-04-15T22:21:41.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:41 smithi092 ceph-mon[29727]: pgmap v139: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:41.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:41 smithi092 ceph-mon[29727]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:21:41.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:41 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:21:41.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:41 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:21:41.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:41 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:41.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:41 smithi092 ceph-mon[24813]: osdmap e41: 6 total, 6 up, 6 in 2024-04-15T22:21:41.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:41 smithi092 ceph-mon[24813]: pgmap v139: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:41.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:41 smithi092 ceph-mon[24813]: from='client.24257 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:21:41.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:41 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:21:41.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:41 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:21:41.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:41 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:42.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[24813]: osdmap e42: 6 total, 6 up, 6 in 2024-04-15T22:21:42.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:42.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:42.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:42.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:42.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:42.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[24813]: pgmap v141: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:42.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:42.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[29727]: osdmap e42: 6 total, 6 up, 6 in 2024-04-15T22:21:42.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:42.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:42.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:42.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:42.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:42.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[29727]: pgmap v141: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:42.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:42 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:42.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:42 smithi176 ceph-mon[25776]: osdmap e42: 6 total, 6 up, 6 in 2024-04-15T22:21:42.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:42 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:42.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:42 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:42.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:42 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:42.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:42 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:42.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:42 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:42.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:42 smithi176 ceph-mon[25776]: pgmap v141: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:42.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:42 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:43.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:43 smithi092 ceph-mon[24813]: osdmap e43: 6 total, 6 up, 6 in 2024-04-15T22:21:43.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:43 smithi092 ceph-mon[24813]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:21:43.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:43 smithi092 ceph-mon[24813]: Cluster is now healthy 2024-04-15T22:21:43.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:43 smithi092 ceph-mon[29727]: osdmap e43: 6 total, 6 up, 6 in 2024-04-15T22:21:43.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:43 smithi092 ceph-mon[29727]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:21:43.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:43 smithi092 ceph-mon[29727]: Cluster is now healthy 2024-04-15T22:21:43.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:43 smithi176 ceph-mon[25776]: osdmap e43: 6 total, 6 up, 6 in 2024-04-15T22:21:43.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:43 smithi176 ceph-mon[25776]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:21:43.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:43 smithi176 ceph-mon[25776]: Cluster is now healthy 2024-04-15T22:21:44.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:44 smithi092 ceph-mon[24813]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:44.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:44 smithi092 ceph-mon[29727]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:44.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:44 smithi176 ceph-mon[25776]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:45.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:45 smithi092 ceph-mon[24813]: from='client.? 172.21.15.176:0/3312285155' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e539da21-061b-4589-b8ff-c5c4e3c9f640"}]: dispatch 2024-04-15T22:21:45.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:45 smithi092 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e539da21-061b-4589-b8ff-c5c4e3c9f640"}]: dispatch 2024-04-15T22:21:45.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:45 smithi092 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e539da21-061b-4589-b8ff-c5c4e3c9f640"}]': finished 2024-04-15T22:21:45.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:45 smithi092 ceph-mon[24813]: osdmap e44: 7 total, 6 up, 7 in 2024-04-15T22:21:45.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:45 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:21:45.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:45 smithi092 ceph-mon[24813]: from='client.? 172.21.15.176:0/1978018554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:21:45.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:45 smithi092 ceph-mon[29727]: from='client.? 172.21.15.176:0/3312285155' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e539da21-061b-4589-b8ff-c5c4e3c9f640"}]: dispatch 2024-04-15T22:21:45.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:45 smithi092 ceph-mon[29727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e539da21-061b-4589-b8ff-c5c4e3c9f640"}]: dispatch 2024-04-15T22:21:45.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:45 smithi092 ceph-mon[29727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e539da21-061b-4589-b8ff-c5c4e3c9f640"}]': finished 2024-04-15T22:21:45.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:45 smithi092 ceph-mon[29727]: osdmap e44: 7 total, 6 up, 7 in 2024-04-15T22:21:45.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:45 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:21:45.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:45 smithi092 ceph-mon[29727]: from='client.? 172.21.15.176:0/1978018554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:21:45.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:45 smithi176 ceph-mon[25776]: from='client.? 172.21.15.176:0/3312285155' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e539da21-061b-4589-b8ff-c5c4e3c9f640"}]: dispatch 2024-04-15T22:21:45.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:45 smithi176 ceph-mon[25776]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e539da21-061b-4589-b8ff-c5c4e3c9f640"}]: dispatch 2024-04-15T22:21:45.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:45 smithi176 ceph-mon[25776]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e539da21-061b-4589-b8ff-c5c4e3c9f640"}]': finished 2024-04-15T22:21:45.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:45 smithi176 ceph-mon[25776]: osdmap e44: 7 total, 6 up, 7 in 2024-04-15T22:21:45.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:45 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:21:45.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:45 smithi176 ceph-mon[25776]: from='client.? 172.21.15.176:0/1978018554' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:21:46.416 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:46 smithi092 ceph-mon[29727]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:46.417 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:46 smithi092 ceph-mon[24813]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:46.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:46 smithi176 ceph-mon[25776]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:48.106 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:21:47 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:21:47] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:21:48.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:48 smithi092 ceph-mon[24813]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:48.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:48 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:48.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:48 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:48.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:48 smithi092 ceph-mon[29727]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:48.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:48.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:48 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:48.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:48 smithi176 ceph-mon[25776]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:48.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:48 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:48.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:48 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:50.326 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:50 smithi176 ceph-mon[25776]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:50.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:50 smithi092 ceph-mon[24813]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:50.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:50 smithi092 ceph-mon[29727]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:52.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:52 smithi176 ceph-mon[25776]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:52.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-15T22:21:52.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:52 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:52.608 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:52 smithi176 ceph-mon[25776]: Deploying daemon osd.6 on smithi176 2024-04-15T22:21:52.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:52 smithi092 ceph-mon[24813]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:52.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-15T22:21:52.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:52 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:52.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:52 smithi092 ceph-mon[24813]: Deploying daemon osd.6 on smithi176 2024-04-15T22:21:52.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:52 smithi092 ceph-mon[29727]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:52.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-15T22:21:52.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:52 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:52.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:52 smithi092 ceph-mon[29727]: Deploying daemon osd.6 on smithi176 2024-04-15T22:21:54.694 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:54 smithi176 ceph-mon[25776]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:54.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:54 smithi092 ceph-mon[24813]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:54.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:54 smithi092 ceph-mon[29727]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:56.589 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:56 smithi176 ceph-mon[25776]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:56.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:56 smithi092 ceph-mon[24813]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:56.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:56 smithi092 ceph-mon[29727]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:57.959 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:57 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:57.959 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:57 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:57.960 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:57 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:57.960 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:57 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:57.960 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:57 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:57.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:57 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:57.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:57 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:57.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:57 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:57.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:57 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:57.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:57 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:57.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:57 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:57.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:57 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:21:57.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:57 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:21:57.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:57 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:21:57.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:57 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:21:58.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:21:58 smithi092 ceph-mon[24813]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:58.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:21:58 smithi092 ceph-mon[29727]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:58.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:21:58 smithi176 ceph-mon[25776]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:21:59.363 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 6 on host 'smithi176' 2024-04-15T22:21:59.982 DEBUG:teuthology.orchestra.run.smithi176:osd.6> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.6.service 2024-04-15T22:21:59.984 INFO:tasks.cephadm:Deploying osd.7 on smithi176 with /dev/vg_nvme/lv_1... 2024-04-15T22:21:59.984 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-15T22:22:00.631 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:00 smithi176 ceph-mon[25776]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:22:00.631 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:00 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:00.632 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:00 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:00.632 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:00 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:00.632 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:00 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:00.632 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:00 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:00.632 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:00 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:00.632 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:00 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:00.632 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:00 smithi176 ceph-mon[25776]: Adjusting osd_memory_target on smithi176 to 5354M 2024-04-15T22:22:00.632 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:00 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:00.632 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:00 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:00.632 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:00 smithi176 ceph-mon[25776]: from='osd.6 [v2:172.21.15.176:6816/2464606803,v1:172.21.15.176:6817/2464606803]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T22:22:00.632 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:00 smithi176 ceph-mon[25776]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T22:22:00.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[24813]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:22:00.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:00.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:00.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:00.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:00.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:00.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:00.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:00.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[24813]: Adjusting osd_memory_target on smithi176 to 5354M 2024-04-15T22:22:00.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:00.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:00.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[24813]: from='osd.6 [v2:172.21.15.176:6816/2464606803,v1:172.21.15.176:6817/2464606803]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T22:22:00.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[24813]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T22:22:00.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[29727]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:22:00.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:00.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:00.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:00.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:00.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:00.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:00.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:00.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[29727]: Adjusting osd_memory_target on smithi176 to 5354M 2024-04-15T22:22:00.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:00.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:00.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[29727]: from='osd.6 [v2:172.21.15.176:6816/2464606803,v1:172.21.15.176:6817/2464606803]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T22:22:00.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:00 smithi092 ceph-mon[29727]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T22:22:01.662 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:01 smithi176 ceph-mon[25776]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T22:22:01.662 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:01 smithi176 ceph-mon[25776]: osdmap e45: 7 total, 6 up, 7 in 2024-04-15T22:22:01.662 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:01 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:01.662 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:01 smithi176 ceph-mon[25776]: from='osd.6 [v2:172.21.15.176:6816/2464606803,v1:172.21.15.176:6817/2464606803]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:22:01.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:01 smithi176 ceph-mon[25776]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:22:01.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:01 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:01.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:01 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:01.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:01 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:01.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:01 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:01.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:01 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:01.663 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:01 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:01.663 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:22:01 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:22:01.419+0000 7f1e2e2a9700 -1 osd.6 0 waiting for initial osdmap 2024-04-15T22:22:01.663 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:22:01 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:22:01.425+0000 7f1e2ac44700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T22:22:01.713 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:22:01] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:22:01.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[29727]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T22:22:01.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[29727]: osdmap e45: 7 total, 6 up, 7 in 2024-04-15T22:22:01.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:01.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[29727]: from='osd.6 [v2:172.21.15.176:6816/2464606803,v1:172.21.15.176:6817/2464606803]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:22:01.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[29727]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:22:01.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:01.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:01.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:01.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:01.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:01.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:01.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[24813]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T22:22:01.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[24813]: osdmap e45: 7 total, 6 up, 7 in 2024-04-15T22:22:01.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:01.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[24813]: from='osd.6 [v2:172.21.15.176:6816/2464606803,v1:172.21.15.176:6817/2464606803]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:22:01.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[24813]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:22:01.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:01.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:01.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:01.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:01.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:01.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:01 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:02.148 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-15T22:22:02.149 INFO:teuthology.orchestra.run.smithi176.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-15T22:22:02.149 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10+0 records in 2024-04-15T22:22:02.149 INFO:teuthology.orchestra.run.smithi176.stderr:10+0 records out 2024-04-15T22:22:02.149 INFO:teuthology.orchestra.run.smithi176.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122163 s, 858 MB/s 2024-04-15T22:22:02.149 INFO:teuthology.orchestra.run.smithi176.stderr:--> Zapping successful for: 2024-04-15T22:22:02.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:02 smithi176 ceph-mon[25776]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:22:02.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:02 smithi176 ceph-mon[25776]: Detected new or changed devices on smithi176 2024-04-15T22:22:02.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:02 smithi176 ceph-mon[25776]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-04-15T22:22:02.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:02 smithi176 ceph-mon[25776]: osdmap e46: 7 total, 6 up, 7 in 2024-04-15T22:22:02.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:02 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:02.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:02 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:02.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:02 smithi176 ceph-mon[25776]: osd.6 [v2:172.21.15.176:6816/2464606803,v1:172.21.15.176:6817/2464606803] boot 2024-04-15T22:22:02.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:02 smithi176 ceph-mon[25776]: osdmap e47: 7 total, 7 up, 7 in 2024-04-15T22:22:02.676 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:02 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:02.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[24813]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:22:02.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[24813]: Detected new or changed devices on smithi176 2024-04-15T22:22:02.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[24813]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-04-15T22:22:02.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[24813]: osdmap e46: 7 total, 6 up, 7 in 2024-04-15T22:22:02.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:02.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:02.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[24813]: osd.6 [v2:172.21.15.176:6816/2464606803,v1:172.21.15.176:6817/2464606803] boot 2024-04-15T22:22:02.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[24813]: osdmap e47: 7 total, 7 up, 7 in 2024-04-15T22:22:02.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:02.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[29727]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T22:22:02.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[29727]: Detected new or changed devices on smithi176 2024-04-15T22:22:02.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[29727]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-04-15T22:22:02.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[29727]: osdmap e46: 7 total, 6 up, 7 in 2024-04-15T22:22:02.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:02.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:02.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[29727]: osd.6 [v2:172.21.15.176:6816/2464606803,v1:172.21.15.176:6817/2464606803] boot 2024-04-15T22:22:02.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[29727]: osdmap e47: 7 total, 7 up, 7 in 2024-04-15T22:22:02.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:02 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:03.419 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:03 smithi176 ceph-mon[25776]: purged_snaps scrub starts 2024-04-15T22:22:03.420 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:03 smithi176 ceph-mon[25776]: purged_snaps scrub ok 2024-04-15T22:22:03.420 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:03 smithi176 ceph-mon[25776]: osdmap e48: 7 total, 7 up, 7 in 2024-04-15T22:22:03.459 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi176:vg_nvme/lv_1 2024-04-15T22:22:03.686 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:03 smithi092 ceph-mon[29727]: purged_snaps scrub starts 2024-04-15T22:22:03.687 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:03 smithi092 ceph-mon[29727]: purged_snaps scrub ok 2024-04-15T22:22:03.687 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:03 smithi092 ceph-mon[29727]: osdmap e48: 7 total, 7 up, 7 in 2024-04-15T22:22:03.687 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:03 smithi092 ceph-mon[24813]: purged_snaps scrub starts 2024-04-15T22:22:03.687 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:03 smithi092 ceph-mon[24813]: purged_snaps scrub ok 2024-04-15T22:22:03.687 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:03 smithi092 ceph-mon[24813]: osdmap e48: 7 total, 7 up, 7 in 2024-04-15T22:22:03.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:22:03] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:22:04.809 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:04 smithi176 ceph-mon[25776]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:04.810 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:04 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:04.810 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:04 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:04.810 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:04 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:04.810 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:04 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:04.810 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:04 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:04.810 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:04 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:04.810 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:04 smithi176 ceph-mon[25776]: osdmap e49: 7 total, 7 up, 7 in 2024-04-15T22:22:04.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[24813]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:04.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:04.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:04.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:04.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:04.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:04.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:04.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[24813]: osdmap e49: 7 total, 7 up, 7 in 2024-04-15T22:22:04.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[29727]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:04.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:04.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:04.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:04.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:04.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:04.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:04.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:04 smithi092 ceph-mon[29727]: osdmap e49: 7 total, 7 up, 7 in 2024-04-15T22:22:06.917 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:06 smithi092 ceph-mon[29727]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:06.918 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:06 smithi092 ceph-mon[29727]: osdmap e50: 7 total, 7 up, 7 in 2024-04-15T22:22:06.918 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:06 smithi092 ceph-mon[24813]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:06.918 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:06 smithi092 ceph-mon[24813]: osdmap e50: 7 total, 7 up, 7 in 2024-04-15T22:22:06.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:06 smithi176 ceph-mon[25776]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:06.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:06 smithi176 ceph-mon[25776]: osdmap e50: 7 total, 7 up, 7 in 2024-04-15T22:22:07.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:07 smithi176 ceph-mon[25776]: osdmap e51: 7 total, 7 up, 7 in 2024-04-15T22:22:07.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:07 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:22:07.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:07 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:22:07.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:07 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:07.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:07 smithi092 ceph-mon[24813]: osdmap e51: 7 total, 7 up, 7 in 2024-04-15T22:22:07.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:07 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:22:07.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:07 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:22:07.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:07 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:07.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:07 smithi092 ceph-mon[29727]: osdmap e51: 7 total, 7 up, 7 in 2024-04-15T22:22:07.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:07 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T22:22:07.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:07 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T22:22:07.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:07 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:08.324 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:07 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:22:07] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:22:08.915 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:08 smithi176 ceph-mon[25776]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:08.915 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:08 smithi176 ceph-mon[25776]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T22:22:08.915 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:08.916 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:08 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:08.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:08 smithi092 ceph-mon[24813]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:08.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:08 smithi092 ceph-mon[24813]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T22:22:08.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:08.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:08 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:08.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:08 smithi092 ceph-mon[29727]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi176:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:08.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:08 smithi092 ceph-mon[29727]: pgmap v163: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T22:22:08.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:08.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:08 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:10.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:10 smithi092 ceph-mon[24813]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-15T22:22:10.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:10 smithi092 ceph-mon[24813]: from='client.? 172.21.15.176:0/4212785335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "024e6cf1-71b7-47a8-a322-edf6e267db5f"}]: dispatch 2024-04-15T22:22:10.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:10 smithi092 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "024e6cf1-71b7-47a8-a322-edf6e267db5f"}]: dispatch 2024-04-15T22:22:10.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:10 smithi092 ceph-mon[24813]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "024e6cf1-71b7-47a8-a322-edf6e267db5f"}]': finished 2024-04-15T22:22:10.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:10 smithi092 ceph-mon[24813]: osdmap e52: 8 total, 7 up, 8 in 2024-04-15T22:22:10.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:10 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:10.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:10 smithi092 ceph-mon[29727]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-15T22:22:10.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:10 smithi092 ceph-mon[29727]: from='client.? 172.21.15.176:0/4212785335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "024e6cf1-71b7-47a8-a322-edf6e267db5f"}]: dispatch 2024-04-15T22:22:10.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:10 smithi092 ceph-mon[29727]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "024e6cf1-71b7-47a8-a322-edf6e267db5f"}]: dispatch 2024-04-15T22:22:10.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:10 smithi092 ceph-mon[29727]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "024e6cf1-71b7-47a8-a322-edf6e267db5f"}]': finished 2024-04-15T22:22:10.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:10 smithi092 ceph-mon[29727]: osdmap e52: 8 total, 7 up, 8 in 2024-04-15T22:22:10.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:10 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:10.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:10 smithi176 ceph-mon[25776]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-04-15T22:22:10.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:10 smithi176 ceph-mon[25776]: from='client.? 172.21.15.176:0/4212785335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "024e6cf1-71b7-47a8-a322-edf6e267db5f"}]: dispatch 2024-04-15T22:22:10.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:10 smithi176 ceph-mon[25776]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "024e6cf1-71b7-47a8-a322-edf6e267db5f"}]: dispatch 2024-04-15T22:22:10.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:10 smithi176 ceph-mon[25776]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "024e6cf1-71b7-47a8-a322-edf6e267db5f"}]': finished 2024-04-15T22:22:10.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:10 smithi176 ceph-mon[25776]: osdmap e52: 8 total, 7 up, 8 in 2024-04-15T22:22:10.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:10 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:11.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:11 smithi092 ceph-mon[24813]: from='client.? 172.21.15.176:0/3425566599' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:22:11.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:11 smithi092 ceph-mon[29727]: from='client.? 172.21.15.176:0/3425566599' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:22:11.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:11 smithi176 ceph-mon[25776]: from='client.? 172.21.15.176:0/3425566599' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T22:22:12.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:12 smithi092 ceph-mon[24813]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T22:22:12.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:12 smithi092 ceph-mon[29727]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T22:22:12.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:12 smithi176 ceph-mon[25776]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T22:22:14.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:14 smithi092 ceph-mon[24813]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:14.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:14 smithi092 ceph-mon[29727]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:14.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:14 smithi176 ceph-mon[25776]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:16.513 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:16 smithi176 ceph-mon[25776]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:16.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:16 smithi092 ceph-mon[24813]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:16.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:16 smithi092 ceph-mon[29727]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:17.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:17 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-15T22:22:17.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:17 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:17.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:17 smithi176 ceph-mon[25776]: Deploying daemon osd.7 on smithi176 2024-04-15T22:22:17.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:17 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-15T22:22:17.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:17 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:17.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:17 smithi092 ceph-mon[24813]: Deploying daemon osd.7 on smithi176 2024-04-15T22:22:17.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:17 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-15T22:22:17.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:17 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:17.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:17 smithi092 ceph-mon[29727]: Deploying daemon osd.7 on smithi176 2024-04-15T22:22:18.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:18 smithi176 ceph-mon[25776]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:18.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:22:18.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:18 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:22:18.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:18 smithi092 ceph-mon[29727]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:18.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:22:18.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:18 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:22:18.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:18 smithi092 ceph-mon[24813]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:18.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:22:18.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:18 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:22:20.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:20 smithi092 ceph-mon[24813]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:20.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:20 smithi092 ceph-mon[29727]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:20.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:20 smithi176 ceph-mon[25776]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:23.043 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:22 smithi176 ceph-mon[25776]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:23.044 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:22 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:23.044 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:22 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:23.044 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:22 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:23.044 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:22 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:23.044 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:22 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:23.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:22 smithi092 ceph-mon[24813]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:23.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:22 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:23.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:22 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:23.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:22 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:23.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:22 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:23.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:22 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:23.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:22 smithi092 ceph-mon[29727]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:23.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:22 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:23.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:22 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:23.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:22 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:23.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:22 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:23.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:22 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:24.213 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:23 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:22:23] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:22:24.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:24 smithi176 ceph-mon[25776]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:24.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:24 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:24.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:24 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:24.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:24 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:24.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:24 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:24.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:24 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:24.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:24 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:25.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[24813]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:25.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:25.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:25.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:25.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:25.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:25.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:25.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[29727]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:25.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:25.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:25.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:25.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:25.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:25.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:24 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:25.714 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:22:25] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:22:25.891 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:25 smithi176 ceph-mon[25776]: from='osd.7 [v2:172.21.15.176:6824/1998364034,v1:172.21.15.176:6825/1998364034]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T22:22:25.891 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:25 smithi176 ceph-mon[25776]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T22:22:25.891 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:25 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:25.891 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:25 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:25.891 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:25 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:25.891 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:25 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:25.892 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:25 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:25.892 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:25 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:26.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[24813]: from='osd.7 [v2:172.21.15.176:6824/1998364034,v1:172.21.15.176:6825/1998364034]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T22:22:26.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[24813]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T22:22:26.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:26.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:26.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:26.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:26.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:26.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:26.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[29727]: from='osd.7 [v2:172.21.15.176:6824/1998364034,v1:172.21.15.176:6825/1998364034]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T22:22:26.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[29727]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T22:22:26.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:26.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:26.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:26.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:26.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:26.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:25 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:26.546 INFO:teuthology.orchestra.run.smithi176.stdout:Created osd(s) 7 on host 'smithi176' 2024-04-15T22:22:27.074 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:27.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-15T22:22:27.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: osdmap e53: 8 total, 7 up, 8 in 2024-04-15T22:22:27.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:27.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='osd.7 [v2:172.21.15.176:6824/1998364034,v1:172.21.15.176:6825/1998364034]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:22:27.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:22:27.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:27.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:27.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:27.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:27.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:27.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:27.075 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:27.076 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:27.076 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:27.076 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:26 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:27.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:27.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-15T22:22:27.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: osdmap e53: 8 total, 7 up, 8 in 2024-04-15T22:22:27.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:27.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='osd.7 [v2:172.21.15.176:6824/1998364034,v1:172.21.15.176:6825/1998364034]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:22:27.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:22:27.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:27.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:27.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:27.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:27.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:27.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:27.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:27.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:27.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:27.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:27.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:27.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-15T22:22:27.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: osdmap e53: 8 total, 7 up, 8 in 2024-04-15T22:22:27.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:27.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='osd.7 [v2:172.21.15.176:6824/1998364034,v1:172.21.15.176:6825/1998364034]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:22:27.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]: dispatch 2024-04-15T22:22:27.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:27.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:27.122 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:27.122 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:27.122 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:27.122 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:27.122 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:27.122 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:27.122 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:27.122 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:26 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:27.634 DEBUG:teuthology.orchestra.run.smithi176:osd.7> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.7.service 2024-04-15T22:22:27.636 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-15T22:22:27.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd stat -f json 2024-04-15T22:22:27.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:27 smithi176 ceph-mon[25776]: Adjusting osd_memory_target on smithi176 to 4016M 2024-04-15T22:22:27.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:27 smithi176 ceph-mon[25776]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-04-15T22:22:27.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:27 smithi176 ceph-mon[25776]: osdmap e54: 8 total, 7 up, 8 in 2024-04-15T22:22:27.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:27 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:27.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:27 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:28.086 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:27 smithi092 ceph-mon[29727]: Adjusting osd_memory_target on smithi176 to 4016M 2024-04-15T22:22:28.087 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:27 smithi092 ceph-mon[29727]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-04-15T22:22:28.087 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:27 smithi092 ceph-mon[29727]: osdmap e54: 8 total, 7 up, 8 in 2024-04-15T22:22:28.087 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:27 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:28.087 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:27 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:28.087 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:27 smithi092 ceph-mon[24813]: Adjusting osd_memory_target on smithi176 to 4016M 2024-04-15T22:22:28.087 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:27 smithi092 ceph-mon[24813]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi176", "root=default"]}]': finished 2024-04-15T22:22:28.087 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:27 smithi092 ceph-mon[24813]: osdmap e54: 8 total, 7 up, 8 in 2024-04-15T22:22:28.087 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:28.087 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:27 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:28.449 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:22:28] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:22:29.085 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[29727]: purged_snaps scrub starts 2024-04-15T22:22:29.085 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[29727]: purged_snaps scrub ok 2024-04-15T22:22:29.085 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[29727]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:29.085 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:29.085 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[29727]: osd.7 [v2:172.21.15.176:6824/1998364034,v1:172.21.15.176:6825/1998364034] boot 2024-04-15T22:22:29.085 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[29727]: osdmap e55: 8 total, 8 up, 8 in 2024-04-15T22:22:29.085 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:29.085 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:29.085 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:29.086 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[24813]: purged_snaps scrub starts 2024-04-15T22:22:29.086 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[24813]: purged_snaps scrub ok 2024-04-15T22:22:29.086 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[24813]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:29.086 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:29.086 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[24813]: osd.7 [v2:172.21.15.176:6824/1998364034,v1:172.21.15.176:6825/1998364034] boot 2024-04-15T22:22:29.086 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[24813]: osdmap e55: 8 total, 8 up, 8 in 2024-04-15T22:22:29.086 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:29.086 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:29.086 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:28 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:29.205 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:28 smithi176 ceph-mon[25776]: purged_snaps scrub starts 2024-04-15T22:22:29.205 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:28 smithi176 ceph-mon[25776]: purged_snaps scrub ok 2024-04-15T22:22:29.205 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:28 smithi176 ceph-mon[25776]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T22:22:29.205 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:29.205 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:28 smithi176 ceph-mon[25776]: osd.7 [v2:172.21.15.176:6824/1998364034,v1:172.21.15.176:6825/1998364034] boot 2024-04-15T22:22:29.205 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:28 smithi176 ceph-mon[25776]: osdmap e55: 8 total, 8 up, 8 in 2024-04-15T22:22:29.205 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:29.205 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:29.205 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:28 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:29.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:29 smithi092 ceph-mon[24813]: osdmap e56: 8 total, 8 up, 8 in 2024-04-15T22:22:29.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:29 smithi092 ceph-mon[29727]: osdmap e56: 8 total, 8 up, 8 in 2024-04-15T22:22:30.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:29 smithi176 ceph-mon[25776]: osdmap e56: 8 total, 8 up, 8 in 2024-04-15T22:22:30.950 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:22:30.950 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:30 smithi092 ceph-mon[24813]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:30.950 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:30 smithi092 ceph-mon[24813]: osdmap e57: 8 total, 8 up, 8 in 2024-04-15T22:22:30.951 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:30 smithi092 ceph-mon[29727]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:30.951 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:30 smithi092 ceph-mon[29727]: osdmap e57: 8 total, 8 up, 8 in 2024-04-15T22:22:31.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:30 smithi176 ceph-mon[25776]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:31.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:30 smithi176 ceph-mon[25776]: osdmap e57: 8 total, 8 up, 8 in 2024-04-15T22:22:31.648 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1713219747,"num_in_osds":8,"osd_in_since":1713219729,"num_remapped_pgs":0} 2024-04-15T22:22:31.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T22:22:31.872 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:22:32.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:31 smithi092 ceph-mon[29727]: osdmap e58: 8 total, 8 up, 8 in 2024-04-15T22:22:32.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:31 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3573271586' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-15T22:22:32.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:31 smithi092 ceph-mon[29727]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T22:22:32.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:31 smithi092 ceph-mon[24813]: osdmap e58: 8 total, 8 up, 8 in 2024-04-15T22:22:32.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:31 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3573271586' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-15T22:22:32.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:31 smithi092 ceph-mon[24813]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T22:22:32.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:31 smithi176 ceph-mon[25776]: osdmap e58: 8 total, 8 up, 8 in 2024-04-15T22:22:32.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:31 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3573271586' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-15T22:22:32.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:31 smithi176 ceph-mon[25776]: pgmap v182: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T22:22:32.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:32 smithi092 ceph-mon[29727]: osdmap e59: 8 total, 8 up, 8 in 2024-04-15T22:22:32.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:32 smithi092 ceph-mon[24813]: osdmap e59: 8 total, 8 up, 8 in 2024-04-15T22:22:33.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:32 smithi176 ceph-mon[25776]: osdmap e59: 8 total, 8 up, 8 in 2024-04-15T22:22:33.632 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:22:33.632 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":59,"fsid":"d537646e-fb75-11ee-bc8f-c7b262605968","created":"2024-04-15T22:17:21.180744+0000","modified":"2024-04-15T22:22:31.817819+0000","last_up_change":"2024-04-15T22:22:27.793462+0000","last_in_change":"2024-04-15T22:22:09.953571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T22:20:27.947110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8ae72c2d-f0aa-4631-9599-82b8a4771c4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3827592669},{"type":"v1","addr":"172.21.15.92:6803","nonce":3827592669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3827592669},{"type":"v1","addr":"172.21.15.92:6805","nonce":3827592669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3827592669},{"type":"v1","addr":"172.21.15.92:6809","nonce":3827592669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3827592669},{"type":"v1","addr":"172.21.15.92:6807","nonce":3827592669}]},"public_addr":"172.21.15.92:6803/3827592669","cluster_addr":"172.21.15.92:6805/3827592669","heartbeat_back_addr":"172.21.15.92:6809/3827592669","heartbeat_front_addr":"172.21.15.92:6807/3827592669","state":["exists","up"]},{"osd":1,"uuid":"aa81f8b4-e851-4b06-bbbc-c5433ec9617a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1540392933},{"type":"v1","addr":"172.21.15.92:6811","nonce":1540392933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1540392933},{"type":"v1","addr":"172.21.15.92:6813","nonce":1540392933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1540392933},{"type":"v1","addr":"172.21.15.92:6817","nonce":1540392933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1540392933},{"type":"v1","addr":"172.21.15.92:6815","nonce":1540392933}]},"public_addr":"172.21.15.92:6811/1540392933","cluster_addr":"172.21.15.92:6813/1540392933","heartbeat_back_addr":"172.21.15.92:6817/1540392933","heartbeat_front_addr":"172.21.15.92:6815/1540392933","state":["exists","up"]},{"osd":2,"uuid":"c726316c-7073-4fc1-b4b1-07dd1f3acb3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":2065441487},{"type":"v1","addr":"172.21.15.92:6819","nonce":2065441487}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":2065441487},{"type":"v1","addr":"172.21.15.92:6821","nonce":2065441487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2065441487},{"type":"v1","addr":"172.21.15.92:6825","nonce":2065441487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":2065441487},{"type":"v1","addr":"172.21.15.92:6823","nonce":2065441487}]},"public_addr":"172.21.15.92:6819/2065441487","cluster_addr":"172.21.15.92:6821/2065441487","heartbeat_back_addr":"172.21.15.92:6825/2065441487","heartbeat_front_addr":"172.21.15.92:6823/2065441487","state":["exists","up"]},{"osd":3,"uuid":"f6d08992-1892-4188-b2d5-39ca9813b173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":3902358626},{"type":"v1","addr":"172.21.15.92:6827","nonce":3902358626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3902358626},{"type":"v1","addr":"172.21.15.92:6829","nonce":3902358626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3902358626},{"type":"v1","addr":"172.21.15.92:6833","nonce":3902358626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":3902358626},{"type":"v1","addr":"172.21.15.92:6831","nonce":3902358626}]},"public_addr":"172.21.15.92:6827/3902358626","cluster_addr":"172.21.15.92:6829/3902358626","heartbeat_back_addr":"172.21.15.92:6833/3902358626","heartbeat_front_addr":"172.21.15.92:6831/3902358626","state":["exists","up"]},{"osd":4,"uuid":"15e0ab27-4f3a-489a-beef-67a3dad86e2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":3350225179},{"type":"v1","addr":"172.21.15.176:6801","nonce":3350225179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":3350225179},{"type":"v1","addr":"172.21.15.176:6803","nonce":3350225179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":3350225179},{"type":"v1","addr":"172.21.15.176:6807","nonce":3350225179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":3350225179},{"type":"v1","addr":"172.21.15.176:6805","nonce":3350225179}]},"public_addr":"172.21.15.176:6801/3350225179","cluster_addr":"172.21.15.176:6803/3350225179","heartbeat_back_addr":"172.21.15.176:6807/3350225179","heartbeat_front_addr":"172.21.15.176:6805/3350225179","state":["exists","up"]},{"osd":5,"uuid":"5d6cda7f-0781-4201-95d2-1fc35a97d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2000788554},{"type":"v1","addr":"172.21.15.176:6809","nonce":2000788554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2000788554},{"type":"v1","addr":"172.21.15.176:6811","nonce":2000788554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2000788554},{"type":"v1","addr":"172.21.15.176:6815","nonce":2000788554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2000788554},{"type":"v1","addr":"172.21.15.176:6813","nonce":2000788554}]},"public_addr":"172.21.15.176:6809/2000788554","cluster_addr":"172.21.15.176:6811/2000788554","heartbeat_back_addr":"172.21.15.176:6815/2000788554","heartbeat_front_addr":"172.21.15.176:6813/2000788554","state":["exists","up"]},{"osd":6,"uuid":"e539da21-061b-4589-b8ff-c5c4e3c9f640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2464606803},{"type":"v1","addr":"172.21.15.176:6817","nonce":2464606803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":2464606803},{"type":"v1","addr":"172.21.15.176:6819","nonce":2464606803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":2464606803},{"type":"v1","addr":"172.21.15.176:6823","nonce":2464606803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":2464606803},{"type":"v1","addr":"172.21.15.176:6821","nonce":2464606803}]},"public_addr":"172.21.15.176:6817/2464606803","cluster_addr":"172.21.15.176:6819/2464606803","heartbeat_back_addr":"172.21.15.176:6823/2464606803","heartbeat_front_addr":"172.21.15.176:6821/2464606803","state":["exists","up"]},{"osd":7,"uuid":"024e6cf1-71b7-47a8-a322-edf6e267db5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1998364034},{"type":"v1","addr":"172.21.15.176:6825","nonce":1998364034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1998364034},{"type":"v1","addr":"172.21.15.176:6827","nonce":1998364034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1998364034},{"type":"v1","addr":"172.21.15.176:6831","nonce":1998364034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1998364034},{"type":"v1","addr":"172.21.15.176:6829","nonce":1998364034}]},"public_addr":"172.21.15.176:6825/1998364034","cluster_addr":"172.21.15.176:6827/1998364034","heartbeat_back_addr":"172.21.15.176:6831/1998364034","heartbeat_front_addr":"172.21.15.176:6829/1998364034","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:19:37.633707+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:20:00.271500+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:20:24.303628+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:20:48.420142+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:21:12.715785+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:21:36.540494+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:22:00.711311+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:22:25.845856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.92:6801/4084236890":"2024-04-16T22:18:17.591288+0000","172.21.15.92:6800/4084236890":"2024-04-16T22:18:17.591288+0000","172.21.15.92:0/2343244098":"2024-04-16T22:18:17.591288+0000","172.21.15.92:0/1349181575":"2024-04-16T22:18:17.591288+0000","172.21.15.92:6800/3203283767":"2024-04-16T22:17:54.738154+0000","172.21.15.92:0/1609306583":"2024-04-16T22:17:54.738154+0000","172.21.15.92:6801/3203283767":"2024-04-16T22:17:54.738154+0000","172.21.15.92:0/459505342":"2024-04-16T22:18:17.591288+0000","172.21.15.92:0/1778304426":"2024-04-16T22:17:54.738154+0000","172.21.15.92:0/3495861683":"2024-04-16T22:17:54.738154+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T22:22:33.925 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:33 smithi092 ceph-mon[29727]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-15T22:22:33.925 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:33 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/726447980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:22:33.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:33 smithi092 ceph-mon[24813]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-15T22:22:33.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:33 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/726447980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:22:34.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:33 smithi176 ceph-mon[25776]: pgmap v184: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-04-15T22:22:34.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:33 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/726447980' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:22:34.343 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T22:20:27.947110+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-15T22:22:34.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-15T22:22:34.558 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:22:36.358 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 1 2024-04-15T22:22:36.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:36 smithi092 ceph-mon[24813]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T22:22:36.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:36 smithi092 ceph-mon[29727]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T22:22:36.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:36 smithi176 ceph-mon[25776]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T22:22:37.022 INFO:tasks.cephadm:Adding prometheus.a on smithi176 2024-04-15T22:22:37.022 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch apply prometheus '1;smithi176=a' 2024-04-15T22:22:37.600 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:37 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/1060710935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T22:22:37.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:37 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/1060710935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T22:22:37.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:37 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/1060710935' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T22:22:38.538 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:38 smithi176 ceph-mon[25776]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-15T22:22:38.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:38 smithi092 ceph-mon[24813]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-15T22:22:38.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:38 smithi092 ceph-mon[29727]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-04-15T22:22:38.922 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled prometheus update... 2024-04-15T22:22:39.514 DEBUG:teuthology.orchestra.run.smithi176:prometheus.a> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@prometheus.a.service 2024-04-15T22:22:39.517 INFO:tasks.cephadm:Adding node-exporter.a on smithi092 2024-04-15T22:22:39.518 INFO:tasks.cephadm:Adding node-exporter.b on smithi176 2024-04-15T22:22:39.518 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch apply node-exporter '2;smithi092=a;smithi176=b' 2024-04-15T22:22:40.010 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[24813]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:40.010 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[24813]: Saving service prometheus spec with placement smithi176=a;count:1 2024-04-15T22:22:40.010 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:40.010 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:40.011 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:40.011 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:40.011 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:40.011 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T22:22:40.011 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[24813]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:40.011 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[29727]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:40.011 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[29727]: Saving service prometheus spec with placement smithi176=a;count:1 2024-04-15T22:22:40.012 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:40.012 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:40.012 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:40.012 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:40.012 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:40.012 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T22:22:40.012 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:39 smithi092 ceph-mon[29727]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:40.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:39 smithi176 ceph-mon[25776]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:40.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:39 smithi176 ceph-mon[25776]: Saving service prometheus spec with placement smithi176=a;count:1 2024-04-15T22:22:40.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:39 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:40.113 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:39 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:40.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:39 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:40.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:39 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:40.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:39 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' 2024-04-15T22:22:40.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:39 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T22:22:40.114 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:39 smithi176 ceph-mon[25776]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:40.114 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:40 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ignoring --setuser ceph since I am not root 2024-04-15T22:22:40.115 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:40 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ignoring --setgroup ceph since I am not root 2024-04-15T22:22:40.115 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:40 smithi176 ceph-mgr[27225]: -- 172.21.15.176:0/3724602260 <== mon.2 v2:172.21.15.176:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e392cea340 con 0x55e392cc3000 2024-04-15T22:22:40.344 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ignoring --setuser ceph since I am not root 2024-04-15T22:22:40.344 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ignoring --setgroup ceph since I am not root 2024-04-15T22:22:40.344 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:40 smithi092 ceph-mgr[25048]: -- 172.21.15.92:0/2923987855 <== mon.0 v2:172.21.15.92:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55849d3ec340 con 0x55849d3c4c00 2024-04-15T22:22:40.714 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:40.482+0000 7f667efb7000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T22:22:40.714 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:40.571+0000 7f667efb7000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T22:22:40.881 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:40 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:40.482+0000 7f878cbe8000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T22:22:40.882 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:40 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:40.573+0000 7f878cbe8000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T22:22:41.167 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:40 smithi176 ceph-mon[25776]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-15T22:22:41.167 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:40 smithi176 ceph-mon[25776]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-15T22:22:41.245 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:40 smithi092 ceph-mon[29727]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-15T22:22:41.245 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:40 smithi092 ceph-mon[29727]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-15T22:22:41.245 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:40 smithi092 ceph-mon[24813]: from='mgr.14150 172.21.15.92:0/3933956414' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-15T22:22:41.245 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:40 smithi092 ceph-mon[24813]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-15T22:22:41.246 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:41 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:41.156+0000 7f667efb7000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T22:22:41.246 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:41 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:41.244+0000 7f667efb7000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T22:22:41.479 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:41.180+0000 7f878cbe8000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T22:22:41.479 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:41.269+0000 7f878cbe8000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T22:22:41.479 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:41.356+0000 7f878cbe8000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T22:22:41.557 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:41 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:41.327+0000 7f667efb7000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T22:22:41.557 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:41 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:41.477+0000 7f667efb7000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T22:22:41.792 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:41.520+0000 7f878cbe8000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T22:22:41.792 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:41.599+0000 7f878cbe8000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T22:22:41.792 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:41.708+0000 7f878cbe8000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T22:22:41.792 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:41.791+0000 7f878cbe8000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T22:22:41.909 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:41 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:41.557+0000 7f667efb7000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T22:22:41.909 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:41 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:41.665+0000 7f667efb7000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T22:22:41.909 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:41 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:41.747+0000 7f667efb7000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T22:22:41.910 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:41 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:41.909+0000 7f667efb7000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T22:22:42.213 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:41 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:41.994+0000 7f667efb7000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T22:22:42.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:41.954+0000 7f878cbe8000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T22:22:42.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:42 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:42.039+0000 7f878cbe8000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T22:22:43.331 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:43.013+0000 7f667efb7000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T22:22:43.331 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:43.141+0000 7f667efb7000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T22:22:43.331 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:43.331+0000 7f667efb7000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T22:22:43.349 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:43.085+0000 7f878cbe8000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T22:22:43.349 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:43.216+0000 7f878cbe8000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T22:22:43.649 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:43.425+0000 7f878cbe8000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T22:22:43.649 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:43.514+0000 7f878cbe8000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T22:22:43.713 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:43.418+0000 7f667efb7000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T22:22:43.714 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:43.555+0000 7f667efb7000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T22:22:43.915 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:43.652+0000 7f878cbe8000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T22:22:44.182 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:43.876+0000 7f667efb7000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T22:22:44.182 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:44 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:44.095+0000 7f667efb7000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T22:22:44.182 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:44 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:44.182+0000 7f667efb7000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T22:22:44.214 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:43.984+0000 7f878cbe8000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T22:22:44.454 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:44 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:44.355+0000 7f667efb7000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T22:22:44.478 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:44 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:44.214+0000 7f878cbe8000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T22:22:44.478 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:44 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:44.302+0000 7f878cbe8000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T22:22:44.764 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:44 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:44.481+0000 7f878cbe8000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T22:22:45.608 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:45 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:45.312+0000 7f667efb7000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T22:22:45.719 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:45 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:45.423+0000 7f878cbe8000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T22:22:45.719 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:45 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:45.718+0000 7f878cbe8000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T22:22:45.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:45 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:45.607+0000 7f667efb7000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T22:22:45.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:45 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:45.703+0000 7f667efb7000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T22:22:45.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:45 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:45.858+0000 7f667efb7000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T22:22:45.971 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:45 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:45.814+0000 7f878cbe8000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T22:22:45.972 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:45 smithi176 ceph-mon[25776]: Active manager daemon y restarted 2024-04-15T22:22:45.972 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:45 smithi176 ceph-mon[25776]: Activating manager daemon y 2024-04-15T22:22:45.972 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:45 smithi176 ceph-mon[25776]: osdmap e60: 8 total, 8 up, 8 in 2024-04-15T22:22:46.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:45 smithi092 ceph-mon[24813]: Active manager daemon y restarted 2024-04-15T22:22:46.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:45 smithi092 ceph-mon[24813]: Activating manager daemon y 2024-04-15T22:22:46.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:45 smithi092 ceph-mon[24813]: osdmap e60: 8 total, 8 up, 8 in 2024-04-15T22:22:46.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:45 smithi092 ceph-mon[29727]: Active manager daemon y restarted 2024-04-15T22:22:46.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:45 smithi092 ceph-mon[29727]: Activating manager daemon y 2024-04-15T22:22:46.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:45 smithi092 ceph-mon[29727]: osdmap e60: 8 total, 8 up, 8 in 2024-04-15T22:22:46.216 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:45 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: [15/Apr/2024:22:22:45] ENGINE Bus STARTING 2024-04-15T22:22:46.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:45 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:45.971+0000 7f878cbe8000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T22:22:46.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:45 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: [15/Apr/2024:22:22:45] ENGINE Bus STARTING 2024-04-15T22:22:46.229 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: CherryPy Checker: 2024-04-15T22:22:46.229 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: The Application mounted at '' has an empty config. 2024-04-15T22:22:46.229 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: 2024-04-15T22:22:46.229 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: [15/Apr/2024:22:22:46] ENGINE Serving on http://:::9283 2024-04-15T22:22:46.229 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: [15/Apr/2024:22:22:46] ENGINE Bus STARTED 2024-04-15T22:22:46.496 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: CherryPy Checker: 2024-04-15T22:22:46.496 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: The Application mounted at '' has an empty config. 2024-04-15T22:22:46.496 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:22:46.496 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: [15/Apr/2024:22:22:46] ENGINE Serving on http://:::9283 2024-04-15T22:22:46.497 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: [15/Apr/2024:22:22:46] ENGINE Bus STARTED 2024-04-15T22:22:46.497 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: [15/Apr/2024:22:22:46] ENGINE Bus STARTING 2024-04-15T22:22:46.764 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: [15/Apr/2024:22:22:46] ENGINE Serving on https://172.21.15.92:7150 2024-04-15T22:22:46.765 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: [15/Apr/2024:22:22:46] ENGINE Bus STARTED 2024-04-15T22:22:46.953 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled node-exporter update... 2024-04-15T22:22:47.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: mgrmap e17: y(active, starting, since 0.059707s), standbys: x 2024-04-15T22:22:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:22:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:22:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:22:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T22:22:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T22:22:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:22:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:22:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:22:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:22:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:22:47.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:47.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:47.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:22:47.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:22:47.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:22:47.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: Manager daemon y is now available 2024-04-15T22:22:47.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:47.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:47.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:47.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:47.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:47.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:47.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:47.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:47.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: mgrmap e17: y(active, starting, since 0.059707s), standbys: x 2024-04-15T22:22:47.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:22:47.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:22:47.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:22:47.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T22:22:47.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T22:22:47.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:22:47.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:22:47.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:22:47.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:47.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:22:47.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:22:47.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:47.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:47.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:22:47.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:22:47.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:22:47.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: Manager daemon y is now available 2024-04-15T22:22:47.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:47.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:47.219 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:47.219 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:47.219 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:47.219 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:47.219 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:47.219 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:47.219 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:47.219 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]': finished 2024-04-15T22:22:47.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]': finished 2024-04-15T22:22:47.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: Standby manager daemon x restarted 2024-04-15T22:22:47.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: Standby manager daemon x started 2024-04-15T22:22:47.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-15T22:22:47.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: Updating smithi176:/etc/ceph/ceph.conf 2024-04-15T22:22:47.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:47.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.? 172.21.15.176:0/3562422871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T22:22:47.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.? 172.21.15.176:0/3562422871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T22:22:47.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.? 172.21.15.176:0/3562422871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T22:22:47.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.? 172.21.15.176:0/3562422871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T22:22:47.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:22:47.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:22:47.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:22:47.221 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:22:47.221 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:47.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:47.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]': finished 2024-04-15T22:22:47.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]': finished 2024-04-15T22:22:47.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: Standby manager daemon x restarted 2024-04-15T22:22:47.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: Standby manager daemon x started 2024-04-15T22:22:47.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-15T22:22:47.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: Updating smithi176:/etc/ceph/ceph.conf 2024-04-15T22:22:47.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:47.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.? 172.21.15.176:0/3562422871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T22:22:47.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.? 172.21.15.176:0/3562422871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T22:22:47.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.? 172.21.15.176:0/3562422871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T22:22:47.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.? 172.21.15.176:0/3562422871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T22:22:47.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:22:47.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:22:47.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:22:47.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:22:47.222 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:47.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: mgrmap e17: y(active, starting, since 0.059707s), standbys: x 2024-04-15T22:22:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T22:22:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T22:22:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T22:22:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T22:22:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T22:22:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T22:22:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T22:22:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T22:22:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T22:22:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T22:22:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T22:22:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T22:22:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T22:22:47.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T22:22:47.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T22:22:47.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T22:22:47.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: Manager daemon y is now available 2024-04-15T22:22:47.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:47.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:47.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:22:47.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:22:47.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:22:47.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:47.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:47.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]: dispatch 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]': finished 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi176", "name": "osd_memory_target"}]': finished 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: Standby manager daemon x restarted 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: Standby manager daemon x started 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: Updating smithi092:/etc/ceph/ceph.conf 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: Updating smithi176:/etc/ceph/ceph.conf 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.? 172.21.15.176:0/3562422871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.? 172.21.15.176:0/3562422871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.? 172.21.15.176:0/3562422871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.? 172.21.15.176:0/3562422871' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:22:47.231 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:22:47.232 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:22:47.232 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:22:47.232 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:47.551 DEBUG:teuthology.orchestra.run.smithi092:node-exporter.a> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@node-exporter.a.service 2024-04-15T22:22:47.553 DEBUG:teuthology.orchestra.run.smithi176:node-exporter.b> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@node-exporter.b.service 2024-04-15T22:22:47.556 INFO:tasks.cephadm:Adding alertmanager.a on smithi092 2024-04-15T22:22:47.556 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch apply alertmanager '1;smithi092=a' 2024-04-15T22:22:48.173 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: [15/Apr/2024:22:22:46] ENGINE Bus STARTING 2024-04-15T22:22:48.174 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: [15/Apr/2024:22:22:46] ENGINE Serving on https://172.21.15.92:7150 2024-04-15T22:22:48.174 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: [15/Apr/2024:22:22:46] ENGINE Bus STARTED 2024-04-15T22:22:48.174 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:22:48.174 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:22:48.174 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: mgrmap e18: y(active, since 1.08004s), standbys: x 2024-04-15T22:22:48.174 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi092=a;smithi176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:48.174 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:48.174 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: Saving service node-exporter spec with placement smithi092=a;smithi176=b;count:2 2024-04-15T22:22:48.174 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:48.174 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:48.174 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:48.174 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:48.174 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T22:22:48.175 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T22:22:48.214 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:22:48 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:22:48] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:22:48.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: [15/Apr/2024:22:22:46] ENGINE Bus STARTING 2024-04-15T22:22:48.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: [15/Apr/2024:22:22:46] ENGINE Serving on https://172.21.15.92:7150 2024-04-15T22:22:48.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: [15/Apr/2024:22:22:46] ENGINE Bus STARTED 2024-04-15T22:22:48.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:22:48.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:22:48.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: mgrmap e18: y(active, since 1.08004s), standbys: x 2024-04-15T22:22:48.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi092=a;smithi176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:48.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:48.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: Saving service node-exporter spec with placement smithi092=a;smithi176=b;count:2 2024-04-15T22:22:48.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:48.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:48.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:48.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:48.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T22:22:48.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T22:22:48.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: [15/Apr/2024:22:22:46] ENGINE Bus STARTING 2024-04-15T22:22:48.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: [15/Apr/2024:22:22:46] ENGINE Serving on https://172.21.15.92:7150 2024-04-15T22:22:48.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: [15/Apr/2024:22:22:46] ENGINE Bus STARTED 2024-04-15T22:22:48.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: Updating smithi176:/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:22:48.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-04-15T22:22:48.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: mgrmap e18: y(active, since 1.08004s), standbys: x 2024-04-15T22:22:48.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi092=a;smithi176=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:48.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:48.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: Saving service node-exporter spec with placement smithi092=a;smithi176=b;count:2 2024-04-15T22:22:48.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:48.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:48.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:48.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:48.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T22:22:48.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T22:22:49.170 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:48 smithi176 ceph-mon[25776]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:49.170 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:48 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-15T22:22:49.170 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:48 smithi176 ceph-mon[25776]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-15T22:22:49.170 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:48 smithi176 ceph-mon[25776]: Deploying daemon prometheus.a on smithi176 2024-04-15T22:22:49.170 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:48 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:49.170 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:48 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:49.309 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:48 smithi092 ceph-mon[24813]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:49.310 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:48 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-15T22:22:49.310 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:48 smithi092 ceph-mon[24813]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-15T22:22:49.310 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:48 smithi092 ceph-mon[24813]: Deploying daemon prometheus.a on smithi176 2024-04-15T22:22:49.310 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:48 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:49.310 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:48 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:49.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:48 smithi092 ceph-mon[29727]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:49.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:48 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-15T22:22:49.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:48 smithi092 ceph-mon[29727]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-15T22:22:49.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:48 smithi092 ceph-mon[29727]: Deploying daemon prometheus.a on smithi176 2024-04-15T22:22:49.311 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:48 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:49.311 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:48 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:49.580 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled alertmanager update... 2024-04-15T22:22:50.198 DEBUG:teuthology.orchestra.run.smithi092:alertmanager.a> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@alertmanager.a.service 2024-04-15T22:22:50.201 INFO:tasks.cephadm:Adding grafana.a on smithi176 2024-04-15T22:22:50.201 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph orch apply grafana '1;smithi176=a' 2024-04-15T22:22:50.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:50 smithi092 ceph-mon[24813]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:50.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:50 smithi092 ceph-mon[24813]: Saving service alertmanager spec with placement smithi092=a;count:1 2024-04-15T22:22:50.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:50 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:50.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:50 smithi092 ceph-mon[24813]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:50.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:50 smithi092 ceph-mon[29727]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:50.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:50 smithi092 ceph-mon[29727]: Saving service alertmanager spec with placement smithi092=a;count:1 2024-04-15T22:22:50.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:50 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:50.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:50 smithi092 ceph-mon[29727]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:50.727 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:50 smithi176 ceph-mon[25776]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:50.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:50 smithi176 ceph-mon[25776]: Saving service alertmanager spec with placement smithi092=a;count:1 2024-04-15T22:22:50.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:50 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:50.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:50 smithi176 ceph-mon[25776]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:53.029 INFO:teuthology.orchestra.run.smithi176.stdout:Scheduled grafana update... 2024-04-15T22:22:53.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:52 smithi176 ceph-mon[25776]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:53.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:52 smithi092 ceph-mon[24813]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:53.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:52 smithi092 ceph-mon[29727]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:54.188 DEBUG:teuthology.orchestra.run.smithi176:grafana.a> sudo journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@grafana.a.service 2024-04-15T22:22:54.191 INFO:tasks.cephadm:Setting up client nodes... 2024-04-15T22:22:54.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-15T22:22:54.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:54 smithi092 ceph-mon[29727]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:54.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:54 smithi092 ceph-mon[29727]: Saving service grafana spec with placement smithi176=a;count:1 2024-04-15T22:22:54.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:54 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:54.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:54 smithi092 ceph-mon[24813]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:54.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:54 smithi092 ceph-mon[24813]: Saving service grafana spec with placement smithi176=a;count:1 2024-04-15T22:22:54.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:54 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:54.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:54 smithi176 ceph-mon[25776]: from='client.24371 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi176=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:22:54.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:54 smithi176 ceph-mon[25776]: Saving service grafana spec with placement smithi176=a;count:1 2024-04-15T22:22:54.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:54 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:22:55.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:55 smithi092 ceph-mon[24813]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:55.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:55 smithi092 ceph-mon[29727]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:55.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:55 smithi176 ceph-mon[25776]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:56.075 INFO:teuthology.orchestra.run.smithi092.stdout:[client.0] 2024-04-15T22:22:56.076 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDAqB1mjrhMBBAA5GdwwMxHcBKsG9WGrcW4Yw== 2024-04-15T22:22:56.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:56 smithi092 ceph-mon[24813]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:56.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:56 smithi092 ceph-mon[29727]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:56.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:56 smithi176 ceph-mon[25776]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:56.812 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-04-15T22:22:56.812 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-15T22:22:56.812 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-15T22:22:56.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-15T22:22:57.177 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:57 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3236310019' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:22:57.177 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:57 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3236310019' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T22:22:57.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:57 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3236310019' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:22:57.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:57 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3236310019' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T22:22:57.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:57 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3236310019' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:22:57.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:57 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3236310019' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T22:22:58.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:22:58 smithi092 ceph-mon[24813]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:58.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:22:58 smithi092 ceph-mon[29727]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:22:58.429 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:22:58 smithi176 ceph-mon[25776]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:00.380 INFO:teuthology.orchestra.run.smithi176.stdout:[client.1] 2024-04-15T22:23:00.381 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQDEqB1mWJpyFhAArLTfnmEmKt1v3WEzziTDdg== 2024-04-15T22:23:00.392 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:00 smithi176 ceph-mon[25776]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:00.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:00 smithi092 ceph-mon[24813]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:00.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:00 smithi092 ceph-mon[29727]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:01.114 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-04-15T22:23:01.114 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-15T22:23:01.114 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-15T22:23:01.162 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-15T22:23:01.162 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-15T22:23:01.162 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph mgr dump --format=json 2024-04-15T22:23:01.346 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:01 smithi176 ceph-mon[25776]: from='client.? 172.21.15.176:0/832024535' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:23:01.346 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:01 smithi176 ceph-mon[25776]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:23:01.347 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:01 smithi176 ceph-mon[25776]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T22:23:01.384 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:01.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:01 smithi092 ceph-mon[29727]: from='client.? 172.21.15.176:0/832024535' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:23:01.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:01 smithi092 ceph-mon[29727]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:23:01.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:01 smithi092 ceph-mon[29727]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T22:23:01.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:01 smithi092 ceph-mon[24813]: from='client.? 172.21.15.176:0/832024535' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:23:01.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:01 smithi092 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T22:23:01.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:01 smithi092 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T22:23:02.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:02 smithi176 ceph-mon[25776]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:02.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:02 smithi092 ceph-mon[24813]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:02.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:02 smithi092 ceph-mon[29727]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:03.286 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:23:03.613 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:03 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/282608407' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-15T22:23:03.641 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:03 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/282608407' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-15T22:23:03.641 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:03 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/282608407' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-15T22:23:03.642 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:23:03] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:23:04.041 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"active_gid":24340,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":2887430743},{"type":"v1","addr":"172.21.15.92:6801","nonce":2887430743}]},"active_addr":"172.21.15.92:6801/2887430743","active_change":"2024-04-15T22:22:45.861918+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24329,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.92:8443/","prometheus":"http://172.21.15.92:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":1560527106}]},{"addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":1678485164}]},{"addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":2867519703}]},{"addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":3040692960}]}]}} 2024-04-15T22:23:04.044 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-15T22:23:04.044 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-15T22:23:04.044 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T22:23:04.262 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:04.633 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:04 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:04.634 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:04 smithi176 ceph-mon[25776]: Detected new or changed devices on smithi176 2024-04-15T22:23:04.634 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:04 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:04.634 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:04 smithi176 ceph-mon[25776]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:04.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:04 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:04.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:04 smithi092 ceph-mon[24813]: Detected new or changed devices on smithi176 2024-04-15T22:23:04.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:04 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:04.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:04 smithi092 ceph-mon[24813]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:04.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:04 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:04.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:04 smithi092 ceph-mon[29727]: Detected new or changed devices on smithi176 2024-04-15T22:23:04.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:04 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:04.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:04 smithi092 ceph-mon[29727]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:04.978 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:04 smithi176 systemd[1]: Starting Ceph prometheus.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:23:05.296 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 podman[42088]: 2024-04-15 22:23:05.166274691 +0000 UTC m=+0.018454690 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-15T22:23:05.549 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 podman[42088]: 2024-04-15 22:23:05.296915591 +0000 UTC m=+0.149095571 container create 2f05fb121fbada94c9b8d5a527c7b22df5cb3285c4112defa9c9c15d5f9403e4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:23:05.549 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 podman[42088]: 2024-04-15 22:23:05.523933038 +0000 UTC m=+0.376113019 container init 2f05fb121fbada94c9b8d5a527c7b22df5cb3285c4112defa9c9c15d5f9403e4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:23:05.549 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 podman[42088]: 2024-04-15 22:23:05.526727728 +0000 UTC m=+0.378907709 container start 2f05fb121fbada94c9b8d5a527c7b22df5cb3285c4112defa9c9c15d5f9403e4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:23:05.828 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.549Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-15T22:23:05.828 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.550Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-15T22:23:05.828 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.550Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-15T22:23:05.828 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.550Z caller=main.go:518 level=info host_details="(Linux 5.14.0-438.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Apr 12 13:55:21 UTC 2024 x86_64 smithi176 (none))" 2024-04-15T22:23:05.828 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.550Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-15T22:23:05.828 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.550Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-15T22:23:05.828 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.551Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-15T22:23:05.828 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.551Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-15T22:23:05.828 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.553Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-15T22:23:05.829 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.555Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-15T22:23:05.829 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.555Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.859µs 2024-04-15T22:23:05.829 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.555Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-15T22:23:05.829 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.555Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-15T22:23:05.829 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.555Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.654µs wal_replay_duration=298.984µs total_replay_duration=342.394µs 2024-04-15T22:23:05.829 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.556Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-15T22:23:05.829 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.556Z caller=main.go:947 level=info msg="TSDB started" 2024-04-15T22:23:05.829 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.556Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-15T22:23:05.829 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.571Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.043507ms db_storage=560ns remote_storage=1.242µs web_handler=294ns query_engine=572ns scrape=242.463µs scrape_sd=14.999µs notify=734ns notify_sd=1.291µs rules=14.572786ms 2024-04-15T22:23:05.829 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:05.571Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-15T22:23:05.829 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 bash[42088]: 2f05fb121fbada94c9b8d5a527c7b22df5cb3285c4112defa9c9c15d5f9403e4 2024-04-15T22:23:05.829 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:05 smithi176 systemd[1]: Started Ceph prometheus.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:23:06.118 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:23:06.119 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":60,"fsid":"d537646e-fb75-11ee-bc8f-c7b262605968","created":"2024-04-15T22:17:21.180744+0000","modified":"2024-04-15T22:22:45.861580+0000","last_up_change":"2024-04-15T22:22:27.793462+0000","last_in_change":"2024-04-15T22:22:09.953571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T22:20:27.947110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8ae72c2d-f0aa-4631-9599-82b8a4771c4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3827592669},{"type":"v1","addr":"172.21.15.92:6803","nonce":3827592669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3827592669},{"type":"v1","addr":"172.21.15.92:6805","nonce":3827592669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3827592669},{"type":"v1","addr":"172.21.15.92:6809","nonce":3827592669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3827592669},{"type":"v1","addr":"172.21.15.92:6807","nonce":3827592669}]},"public_addr":"172.21.15.92:6803/3827592669","cluster_addr":"172.21.15.92:6805/3827592669","heartbeat_back_addr":"172.21.15.92:6809/3827592669","heartbeat_front_addr":"172.21.15.92:6807/3827592669","state":["exists","up"]},{"osd":1,"uuid":"aa81f8b4-e851-4b06-bbbc-c5433ec9617a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1540392933},{"type":"v1","addr":"172.21.15.92:6811","nonce":1540392933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1540392933},{"type":"v1","addr":"172.21.15.92:6813","nonce":1540392933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1540392933},{"type":"v1","addr":"172.21.15.92:6817","nonce":1540392933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1540392933},{"type":"v1","addr":"172.21.15.92:6815","nonce":1540392933}]},"public_addr":"172.21.15.92:6811/1540392933","cluster_addr":"172.21.15.92:6813/1540392933","heartbeat_back_addr":"172.21.15.92:6817/1540392933","heartbeat_front_addr":"172.21.15.92:6815/1540392933","state":["exists","up"]},{"osd":2,"uuid":"c726316c-7073-4fc1-b4b1-07dd1f3acb3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":2065441487},{"type":"v1","addr":"172.21.15.92:6819","nonce":2065441487}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":2065441487},{"type":"v1","addr":"172.21.15.92:6821","nonce":2065441487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2065441487},{"type":"v1","addr":"172.21.15.92:6825","nonce":2065441487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":2065441487},{"type":"v1","addr":"172.21.15.92:6823","nonce":2065441487}]},"public_addr":"172.21.15.92:6819/2065441487","cluster_addr":"172.21.15.92:6821/2065441487","heartbeat_back_addr":"172.21.15.92:6825/2065441487","heartbeat_front_addr":"172.21.15.92:6823/2065441487","state":["exists","up"]},{"osd":3,"uuid":"f6d08992-1892-4188-b2d5-39ca9813b173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":3902358626},{"type":"v1","addr":"172.21.15.92:6827","nonce":3902358626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3902358626},{"type":"v1","addr":"172.21.15.92:6829","nonce":3902358626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3902358626},{"type":"v1","addr":"172.21.15.92:6833","nonce":3902358626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":3902358626},{"type":"v1","addr":"172.21.15.92:6831","nonce":3902358626}]},"public_addr":"172.21.15.92:6827/3902358626","cluster_addr":"172.21.15.92:6829/3902358626","heartbeat_back_addr":"172.21.15.92:6833/3902358626","heartbeat_front_addr":"172.21.15.92:6831/3902358626","state":["exists","up"]},{"osd":4,"uuid":"15e0ab27-4f3a-489a-beef-67a3dad86e2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":3350225179},{"type":"v1","addr":"172.21.15.176:6801","nonce":3350225179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":3350225179},{"type":"v1","addr":"172.21.15.176:6803","nonce":3350225179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":3350225179},{"type":"v1","addr":"172.21.15.176:6807","nonce":3350225179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":3350225179},{"type":"v1","addr":"172.21.15.176:6805","nonce":3350225179}]},"public_addr":"172.21.15.176:6801/3350225179","cluster_addr":"172.21.15.176:6803/3350225179","heartbeat_back_addr":"172.21.15.176:6807/3350225179","heartbeat_front_addr":"172.21.15.176:6805/3350225179","state":["exists","up"]},{"osd":5,"uuid":"5d6cda7f-0781-4201-95d2-1fc35a97d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2000788554},{"type":"v1","addr":"172.21.15.176:6809","nonce":2000788554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2000788554},{"type":"v1","addr":"172.21.15.176:6811","nonce":2000788554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2000788554},{"type":"v1","addr":"172.21.15.176:6815","nonce":2000788554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2000788554},{"type":"v1","addr":"172.21.15.176:6813","nonce":2000788554}]},"public_addr":"172.21.15.176:6809/2000788554","cluster_addr":"172.21.15.176:6811/2000788554","heartbeat_back_addr":"172.21.15.176:6815/2000788554","heartbeat_front_addr":"172.21.15.176:6813/2000788554","state":["exists","up"]},{"osd":6,"uuid":"e539da21-061b-4589-b8ff-c5c4e3c9f640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2464606803},{"type":"v1","addr":"172.21.15.176:6817","nonce":2464606803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":2464606803},{"type":"v1","addr":"172.21.15.176:6819","nonce":2464606803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":2464606803},{"type":"v1","addr":"172.21.15.176:6823","nonce":2464606803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":2464606803},{"type":"v1","addr":"172.21.15.176:6821","nonce":2464606803}]},"public_addr":"172.21.15.176:6817/2464606803","cluster_addr":"172.21.15.176:6819/2464606803","heartbeat_back_addr":"172.21.15.176:6823/2464606803","heartbeat_front_addr":"172.21.15.176:6821/2464606803","state":["exists","up"]},{"osd":7,"uuid":"024e6cf1-71b7-47a8-a322-edf6e267db5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1998364034},{"type":"v1","addr":"172.21.15.176:6825","nonce":1998364034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1998364034},{"type":"v1","addr":"172.21.15.176:6827","nonce":1998364034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1998364034},{"type":"v1","addr":"172.21.15.176:6831","nonce":1998364034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1998364034},{"type":"v1","addr":"172.21.15.176:6829","nonce":1998364034}]},"public_addr":"172.21.15.176:6825/1998364034","cluster_addr":"172.21.15.176:6827/1998364034","heartbeat_back_addr":"172.21.15.176:6831/1998364034","heartbeat_front_addr":"172.21.15.176:6829/1998364034","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:19:37.633707+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:20:00.271500+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:20:24.303628+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:20:48.420142+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:21:12.715785+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:21:36.540494+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:22:00.711311+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:22:25.845856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.92:6801/4139644651":"2024-04-16T22:22:45.861533+0000","172.21.15.92:0/1886766826":"2024-04-16T22:22:45.861533+0000","172.21.15.92:0/396238662":"2024-04-16T22:22:45.861533+0000","172.21.15.92:0/379636010":"2024-04-16T22:22:45.861533+0000","172.21.15.92:6801/4084236890":"2024-04-16T22:18:17.591288+0000","172.21.15.92:6800/4084236890":"2024-04-16T22:18:17.591288+0000","172.21.15.92:0/2343244098":"2024-04-16T22:18:17.591288+0000","172.21.15.92:6800/4139644651":"2024-04-16T22:22:45.861533+0000","172.21.15.92:0/1349181575":"2024-04-16T22:18:17.591288+0000","172.21.15.92:6800/3203283767":"2024-04-16T22:17:54.738154+0000","172.21.15.92:0/3300011615":"2024-04-16T22:22:45.861533+0000","172.21.15.92:0/1609306583":"2024-04-16T22:17:54.738154+0000","172.21.15.92:6801/3203283767":"2024-04-16T22:17:54.738154+0000","172.21.15.92:0/459505342":"2024-04-16T22:18:17.591288+0000","172.21.15.92:0/1778304426":"2024-04-16T22:17:54.738154+0000","172.21.15.92:0/3495861683":"2024-04-16T22:17:54.738154+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T22:23:06.704 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:06 smithi092 systemd[1]: Starting Ceph node-exporter.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:23:06.908 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-15T22:23:06.908 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T22:23:06.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:06 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:06.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:06 smithi092 ceph-mon[29727]: Deploying daemon node-exporter.a on smithi092 2024-04-15T22:23:06.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:06 smithi092 ceph-mon[29727]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:06.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:06 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:06.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:06 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2998898840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:23:06.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:06 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:06.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:06 smithi092 ceph-mon[24813]: Deploying daemon node-exporter.a on smithi092 2024-04-15T22:23:06.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:06 smithi092 ceph-mon[24813]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:06.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:06 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:06.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:06 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2998898840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:23:06.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:06 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:06.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:06 smithi176 ceph-mon[25776]: Deploying daemon node-exporter.a on smithi092 2024-04-15T22:23:06.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:06 smithi176 ceph-mon[25776]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:06.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:06 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:06.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:06 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2998898840' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:23:07.157 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:07.365 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:07 smithi092 bash[45837]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-15T22:23:07.714 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:07 smithi092 bash[45837]: Getting image source signatures 2024-04-15T22:23:07.714 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:07 smithi092 bash[45837]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-15T22:23:07.714 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:07 smithi092 bash[45837]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-15T22:23:07.714 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:07 smithi092 bash[45837]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-15T22:23:08.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:08 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:23:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:23:09.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:08 smithi176 ceph-mon[25776]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:09.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:08 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:09.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:08 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:09.300 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:08 smithi092 ceph-mon[24813]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:09.300 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:08 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:09.300 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:08 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:09.300 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:08 smithi092 ceph-mon[29727]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:09.300 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:08 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:09.300 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:08 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:09.629 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:09 smithi092 bash[45837]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-15T22:23:09.912 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:09 smithi092 bash[45837]: Writing manifest to image destination 2024-04-15T22:23:09.973 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:23:09.973 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":60,"fsid":"d537646e-fb75-11ee-bc8f-c7b262605968","created":"2024-04-15T22:17:21.180744+0000","modified":"2024-04-15T22:22:45.861580+0000","last_up_change":"2024-04-15T22:22:27.793462+0000","last_in_change":"2024-04-15T22:22:09.953571+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T22:20:27.947110+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"8ae72c2d-f0aa-4631-9599-82b8a4771c4f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":3827592669},{"type":"v1","addr":"172.21.15.92:6803","nonce":3827592669}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":3827592669},{"type":"v1","addr":"172.21.15.92:6805","nonce":3827592669}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":3827592669},{"type":"v1","addr":"172.21.15.92:6809","nonce":3827592669}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":3827592669},{"type":"v1","addr":"172.21.15.92:6807","nonce":3827592669}]},"public_addr":"172.21.15.92:6803/3827592669","cluster_addr":"172.21.15.92:6805/3827592669","heartbeat_back_addr":"172.21.15.92:6809/3827592669","heartbeat_front_addr":"172.21.15.92:6807/3827592669","state":["exists","up"]},{"osd":1,"uuid":"aa81f8b4-e851-4b06-bbbc-c5433ec9617a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":1540392933},{"type":"v1","addr":"172.21.15.92:6811","nonce":1540392933}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":1540392933},{"type":"v1","addr":"172.21.15.92:6813","nonce":1540392933}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":1540392933},{"type":"v1","addr":"172.21.15.92:6817","nonce":1540392933}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":1540392933},{"type":"v1","addr":"172.21.15.92:6815","nonce":1540392933}]},"public_addr":"172.21.15.92:6811/1540392933","cluster_addr":"172.21.15.92:6813/1540392933","heartbeat_back_addr":"172.21.15.92:6817/1540392933","heartbeat_front_addr":"172.21.15.92:6815/1540392933","state":["exists","up"]},{"osd":2,"uuid":"c726316c-7073-4fc1-b4b1-07dd1f3acb3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":2065441487},{"type":"v1","addr":"172.21.15.92:6819","nonce":2065441487}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6820","nonce":2065441487},{"type":"v1","addr":"172.21.15.92:6821","nonce":2065441487}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6824","nonce":2065441487},{"type":"v1","addr":"172.21.15.92:6825","nonce":2065441487}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":2065441487},{"type":"v1","addr":"172.21.15.92:6823","nonce":2065441487}]},"public_addr":"172.21.15.92:6819/2065441487","cluster_addr":"172.21.15.92:6821/2065441487","heartbeat_back_addr":"172.21.15.92:6825/2065441487","heartbeat_front_addr":"172.21.15.92:6823/2065441487","state":["exists","up"]},{"osd":3,"uuid":"f6d08992-1892-4188-b2d5-39ca9813b173","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6826","nonce":3902358626},{"type":"v1","addr":"172.21.15.92:6827","nonce":3902358626}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":3902358626},{"type":"v1","addr":"172.21.15.92:6829","nonce":3902358626}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6832","nonce":3902358626},{"type":"v1","addr":"172.21.15.92:6833","nonce":3902358626}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":3902358626},{"type":"v1","addr":"172.21.15.92:6831","nonce":3902358626}]},"public_addr":"172.21.15.92:6827/3902358626","cluster_addr":"172.21.15.92:6829/3902358626","heartbeat_back_addr":"172.21.15.92:6833/3902358626","heartbeat_front_addr":"172.21.15.92:6831/3902358626","state":["exists","up"]},{"osd":4,"uuid":"15e0ab27-4f3a-489a-beef-67a3dad86e2f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":3350225179},{"type":"v1","addr":"172.21.15.176:6801","nonce":3350225179}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":3350225179},{"type":"v1","addr":"172.21.15.176:6803","nonce":3350225179}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":3350225179},{"type":"v1","addr":"172.21.15.176:6807","nonce":3350225179}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":3350225179},{"type":"v1","addr":"172.21.15.176:6805","nonce":3350225179}]},"public_addr":"172.21.15.176:6801/3350225179","cluster_addr":"172.21.15.176:6803/3350225179","heartbeat_back_addr":"172.21.15.176:6807/3350225179","heartbeat_front_addr":"172.21.15.176:6805/3350225179","state":["exists","up"]},{"osd":5,"uuid":"5d6cda7f-0781-4201-95d2-1fc35a97d7ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2000788554},{"type":"v1","addr":"172.21.15.176:6809","nonce":2000788554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2000788554},{"type":"v1","addr":"172.21.15.176:6811","nonce":2000788554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6814","nonce":2000788554},{"type":"v1","addr":"172.21.15.176:6815","nonce":2000788554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":2000788554},{"type":"v1","addr":"172.21.15.176:6813","nonce":2000788554}]},"public_addr":"172.21.15.176:6809/2000788554","cluster_addr":"172.21.15.176:6811/2000788554","heartbeat_back_addr":"172.21.15.176:6815/2000788554","heartbeat_front_addr":"172.21.15.176:6813/2000788554","state":["exists","up"]},{"osd":6,"uuid":"e539da21-061b-4589-b8ff-c5c4e3c9f640","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6816","nonce":2464606803},{"type":"v1","addr":"172.21.15.176:6817","nonce":2464606803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6818","nonce":2464606803},{"type":"v1","addr":"172.21.15.176:6819","nonce":2464606803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6822","nonce":2464606803},{"type":"v1","addr":"172.21.15.176:6823","nonce":2464606803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6820","nonce":2464606803},{"type":"v1","addr":"172.21.15.176:6821","nonce":2464606803}]},"public_addr":"172.21.15.176:6817/2464606803","cluster_addr":"172.21.15.176:6819/2464606803","heartbeat_back_addr":"172.21.15.176:6823/2464606803","heartbeat_front_addr":"172.21.15.176:6821/2464606803","state":["exists","up"]},{"osd":7,"uuid":"024e6cf1-71b7-47a8-a322-edf6e267db5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6824","nonce":1998364034},{"type":"v1","addr":"172.21.15.176:6825","nonce":1998364034}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6826","nonce":1998364034},{"type":"v1","addr":"172.21.15.176:6827","nonce":1998364034}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6830","nonce":1998364034},{"type":"v1","addr":"172.21.15.176:6831","nonce":1998364034}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6828","nonce":1998364034},{"type":"v1","addr":"172.21.15.176:6829","nonce":1998364034}]},"public_addr":"172.21.15.176:6825/1998364034","cluster_addr":"172.21.15.176:6827/1998364034","heartbeat_back_addr":"172.21.15.176:6831/1998364034","heartbeat_front_addr":"172.21.15.176:6829/1998364034","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:19:37.633707+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:20:00.271500+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:20:24.303628+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:20:48.420142+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:21:12.715785+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:21:36.540494+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:22:00.711311+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T22:22:25.845856+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.92:6801/4139644651":"2024-04-16T22:22:45.861533+0000","172.21.15.92:0/1886766826":"2024-04-16T22:22:45.861533+0000","172.21.15.92:0/396238662":"2024-04-16T22:22:45.861533+0000","172.21.15.92:0/379636010":"2024-04-16T22:22:45.861533+0000","172.21.15.92:6801/4084236890":"2024-04-16T22:18:17.591288+0000","172.21.15.92:6800/4084236890":"2024-04-16T22:18:17.591288+0000","172.21.15.92:0/2343244098":"2024-04-16T22:18:17.591288+0000","172.21.15.92:6800/4139644651":"2024-04-16T22:22:45.861533+0000","172.21.15.92:0/1349181575":"2024-04-16T22:18:17.591288+0000","172.21.15.92:6800/3203283767":"2024-04-16T22:17:54.738154+0000","172.21.15.92:0/3300011615":"2024-04-16T22:22:45.861533+0000","172.21.15.92:0/1609306583":"2024-04-16T22:17:54.738154+0000","172.21.15.92:6801/3203283767":"2024-04-16T22:17:54.738154+0000","172.21.15.92:0/459505342":"2024-04-16T22:18:17.591288+0000","172.21.15.92:0/1778304426":"2024-04-16T22:17:54.738154+0000","172.21.15.92:0/3495861683":"2024-04-16T22:17:54.738154+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T22:23:10.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:10 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2816108721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:23:10.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:10 smithi092 ceph-mon[29727]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:10.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:10 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2816108721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:23:10.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:10 smithi092 ceph-mon[24813]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:10.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:10 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2816108721' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T22:23:10.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:10 smithi176 ceph-mon[25776]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:10.522 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:10 smithi092 podman[45837]: 2024-04-15 22:23:10.253719894 +0000 UTC m=+3.275956865 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-15T22:23:10.523 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:10 smithi092 podman[45837]: 2024-04-15 22:23:10.385999416 +0000 UTC m=+3.408236386 container create 2b352622d41eb20126c2c102fbd09e9e826531a8e44c23ae4fd4a3dc72b08017 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-15T22:23:10.523 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:10 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:23:10] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:23:10.963 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:10 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:23:10] "GET /metrics HTTP/1.1" 200 191509 "" "Prometheus/2.33.4" 2024-04-15T22:23:10.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-15T22:23:10.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-15T22:23:10.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-15T22:23:10.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-15T22:23:10.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-15T22:23:10.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-15T22:23:10.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-15T22:23:10.993 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-15T22:23:11.454 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:11 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:11.454 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:11 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:11.559 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:11.559 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:11.560 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:11.560 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:11.560 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 podman[45837]: 2024-04-15 22:23:11.267580092 +0000 UTC m=+4.289817071 container init 2b352622d41eb20126c2c102fbd09e9e826531a8e44c23ae4fd4a3dc72b08017 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-15T22:23:11.560 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 podman[45837]: 2024-04-15 22:23:11.271980743 +0000 UTC m=+4.294217722 container start 2b352622d41eb20126c2c102fbd09e9e826531a8e44c23ae4fd4a3dc72b08017 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-15T22:23:11.560 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.284Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.284Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.284Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.284Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=arp 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=edac 2024-04-15T22:23:11.561 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.286Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-15T22:23:11.562 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=os 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=stat 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=time 2024-04-15T22:23:11.563 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-15T22:23:11.564 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=uname 2024-04-15T22:23:11.564 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-15T22:23:11.564 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-15T22:23:11.564 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-15T22:23:11.564 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.287Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-15T22:23:11.564 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a[46051]: ts=2024-04-15T22:23:11.288Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-15T22:23:11.564 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 bash[45837]: 2b352622d41eb20126c2c102fbd09e9e826531a8e44c23ae4fd4a3dc72b08017 2024-04-15T22:23:11.564 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:23:11 smithi092 systemd[1]: Started Ceph node-exporter.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:23:11.795 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:12.170 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:23:11] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:23:12.383 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:12 smithi176 systemd[1]: Starting Ceph node-exporter.b for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:23:12.676 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:12.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:12 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:12.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:12 smithi092 ceph-mon[24813]: Deploying daemon node-exporter.b on smithi176 2024-04-15T22:23:12.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:12 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:12.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:12 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:12.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:12 smithi092 ceph-mon[24813]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:12.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:12 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:12.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:12 smithi092 ceph-mon[29727]: Deploying daemon node-exporter.b on smithi176 2024-04-15T22:23:12.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:12 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:12.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:12 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:12.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:12 smithi092 ceph-mon[29727]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:12.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:12 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:12.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:12 smithi176 ceph-mon[25776]: Deploying daemon node-exporter.b on smithi176 2024-04-15T22:23:12.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:12 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:12.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:12 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:12.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:12 smithi176 ceph-mon[25776]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:12.729 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:12 smithi176 bash[42900]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-15T22:23:13.478 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:13 smithi176 bash[42900]: Getting image source signatures 2024-04-15T22:23:13.479 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:13 smithi176 bash[42900]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-15T22:23:13.479 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:13 smithi176 bash[42900]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-15T22:23:13.479 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:13 smithi176 bash[42900]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-15T22:23:14.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:23:13 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:23:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:23:15.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:14 smithi092 ceph-mon[24813]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:15.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:14 smithi092 ceph-mon[29727]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:15.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:14 smithi176 ceph-mon[25776]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:15.479 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 bash[42900]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-15T22:23:15.479 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 bash[42900]: Writing manifest to image destination 2024-04-15T22:23:15.757 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:15.757 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:15.758 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:15.762 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:15.762 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:15.762 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:15.798 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 podman[42900]: 2024-04-15 22:23:15.473233323 +0000 UTC m=+3.184446842 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-15T22:23:15.798 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 podman[42900]: 2024-04-15 22:23:15.571735381 +0000 UTC m=+3.282948886 container create 774c4598b14ff40483d2860ca79091ee0904043353c9db21596788c5f5f81b01 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-15T22:23:16.079 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 podman[42900]: 2024-04-15 22:23:15.798829314 +0000 UTC m=+3.510042824 container init 774c4598b14ff40483d2860ca79091ee0904043353c9db21596788c5f5f81b01 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-15T22:23:16.080 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 podman[42900]: 2024-04-15 22:23:15.802075168 +0000 UTC m=+3.513288674 container start 774c4598b14ff40483d2860ca79091ee0904043353c9db21596788c5f5f81b01 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-15T22:23:16.080 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.806Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-15T22:23:16.080 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.806Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-15T22:23:16.080 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-15T22:23:16.080 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-15T22:23:16.080 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-15T22:23:16.080 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=arp 2024-04-15T22:23:16.080 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-15T22:23:16.080 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-15T22:23:16.080 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-15T22:23:16.081 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-15T22:23:16.081 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-15T22:23:16.081 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-15T22:23:16.081 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-15T22:23:16.081 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-15T22:23:16.081 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=edac 2024-04-15T22:23:16.082 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-15T22:23:16.082 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-15T22:23:16.082 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-15T22:23:16.082 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-15T22:23:16.082 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-15T22:23:16.082 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-15T22:23:16.082 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-15T22:23:16.082 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-15T22:23:16.082 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-15T22:23:16.082 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-15T22:23:16.082 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-15T22:23:16.083 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-15T22:23:16.083 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-15T22:23:16.083 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-15T22:23:16.083 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-15T22:23:16.083 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-15T22:23:16.083 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=os 2024-04-15T22:23:16.083 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-15T22:23:16.083 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-15T22:23:16.083 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-15T22:23:16.083 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-15T22:23:16.083 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-15T22:23:16.083 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-15T22:23:16.084 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.807Z caller=node_exporter.go:115 level=info collector=stat 2024-04-15T22:23:16.084 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.808Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-15T22:23:16.084 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.808Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-15T22:23:16.084 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.808Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-15T22:23:16.084 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.808Z caller=node_exporter.go:115 level=info collector=time 2024-04-15T22:23:16.084 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.808Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-15T22:23:16.084 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.808Z caller=node_exporter.go:115 level=info collector=uname 2024-04-15T22:23:16.084 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.808Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-15T22:23:16.084 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.808Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-15T22:23:16.084 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.808Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-15T22:23:16.084 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.808Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-15T22:23:16.085 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b[42951]: ts=2024-04-15T22:23:15.808Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-15T22:23:16.085 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 bash[42900]: 774c4598b14ff40483d2860ca79091ee0904043353c9db21596788c5f5f81b01 2024-04-15T22:23:16.085 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:23:15 smithi176 systemd[1]: Started Ceph node-exporter.b for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:23:17.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[24813]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:17.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:17.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:17.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:17.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:17.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:17.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:17.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[24813]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:23:17.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[24813]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-15T22:23:17.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[24813]: Reconfiguring daemon prometheus.a on smithi176 2024-04-15T22:23:17.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[29727]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:17.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:17.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:17.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:17.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:17.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:17.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:17.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[29727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:23:17.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[29727]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-15T22:23:17.216 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:16 smithi092 ceph-mon[29727]: Reconfiguring daemon prometheus.a on smithi176 2024-04-15T22:23:17.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:16 smithi176 ceph-mon[25776]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:17.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:16 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:17.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:16 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:17.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:16 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:17.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:16 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:17.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:16 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:17.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:16 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:17.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:16 smithi176 ceph-mon[25776]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:23:17.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:16 smithi176 ceph-mon[25776]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-15T22:23:17.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:16 smithi176 ceph-mon[25776]: Reconfiguring daemon prometheus.a on smithi176 2024-04-15T22:23:19.152 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:18 smithi092 ceph-mon[29727]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:19.152 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:18 smithi092 ceph-mon[24813]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:19.166 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:18 smithi176 ceph-mon[25776]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:19.177 INFO:teuthology.orchestra.run.smithi092.stdout:111669149727 2024-04-15T22:23:19.177 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-15T22:23:19.463 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:19 smithi176 systemd[1]: Stopping Ceph prometheus.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:23:19.728 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:19 smithi176 bash[43192]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus.a" found: no such container 2024-04-15T22:23:20.137 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:20.023Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-15T22:23:20.137 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:20.023Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-15T22:23:20.138 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:20.023Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-15T22:23:20.138 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:20.023Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-15T22:23:20.138 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:20.024Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-15T22:23:20.138 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:20.023Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-15T22:23:20.138 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:20.024Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-15T22:23:20.138 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:20.024Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-15T22:23:20.138 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:20.024Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-15T22:23:20.138 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:20.024Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-15T22:23:20.138 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:20.024Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-15T22:23:20.138 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[42104]: ts=2024-04-15T22:23:20.024Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-15T22:23:20.138 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 podman[43211]: 2024-04-15 22:23:20.034378018 +0000 UTC m=+0.560680198 container died 2f05fb121fbada94c9b8d5a527c7b22df5cb3285c4112defa9c9c15d5f9403e4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:23:20.422 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:20 smithi176 ceph-mon[25776]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:20.441 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:20 smithi092 ceph-mon[24813]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:20.442 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:20 smithi092 ceph-mon[29727]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:20.728 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 podman[43211]: 2024-04-15 22:23:20.423653987 +0000 UTC m=+0.949956172 container cleanup 2f05fb121fbada94c9b8d5a527c7b22df5cb3285c4112defa9c9c15d5f9403e4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:23:20.728 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 bash[43211]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a 2024-04-15T22:23:20.728 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:20 smithi176 bash[43273]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus.a" found: no such container 2024-04-15T22:23:21.379 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 podman[43260]: 2024-04-15 22:23:21.095922105 +0000 UTC m=+1.064629067 container remove 2f05fb121fbada94c9b8d5a527c7b22df5cb3285c4112defa9c9c15d5f9403e4 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:23:21.379 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-15T22:23:21.379 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 systemd[1]: Stopped Ceph prometheus.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:23:21.379 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 systemd[1]: Starting Ceph prometheus.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:23:21.379 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 podman[43353]: 2024-04-15 22:23:21.242422596 +0000 UTC m=+0.018803564 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-15T22:23:21.650 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 podman[43353]: 2024-04-15 22:23:21.380185603 +0000 UTC m=+0.156566558 container create 69c7526eb7559cbab644bdbcd9c6d0e77583ea052eb89f807f5340d5399756ce (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:23:21.651 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 podman[43353]: 2024-04-15 22:23:21.623943443 +0000 UTC m=+0.400324398 container init 69c7526eb7559cbab644bdbcd9c6d0e77583ea052eb89f807f5340d5399756ce (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:23:21.651 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 podman[43353]: 2024-04-15 22:23:21.627382866 +0000 UTC m=+0.403763822 container start 69c7526eb7559cbab644bdbcd9c6d0e77583ea052eb89f807f5340d5399756ce (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:23:21.979 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.651Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-15T22:23:21.979 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.651Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-15T22:23:21.979 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.651Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-15T22:23:21.979 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.651Z caller=main.go:518 level=info host_details="(Linux 5.14.0-438.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Apr 12 13:55:21 UTC 2024 x86_64 smithi176 (none))" 2024-04-15T22:23:21.979 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.651Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-15T22:23:21.979 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.651Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.653Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.654Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.655Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.656Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.656Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.472µs 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.656Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 bash[43353]: 69c7526eb7559cbab644bdbcd9c6d0e77583ea052eb89f807f5340d5399756ce 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 systemd[1]: Started Ceph prometheus.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.870Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.871Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.871Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=39.725µs wal_replay_duration=214.163858ms total_replay_duration=214.225932ms 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.872Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.872Z caller=main.go:947 level=info msg="TSDB started" 2024-04-15T22:23:21.980 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.872Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-15T22:23:21.981 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.888Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.814919ms db_storage=668ns remote_storage=1.641µs web_handler=318ns query_engine=584ns scrape=294.803µs scrape_sd=22.864µs notify=995ns notify_sd=1.888µs rules=15.293628ms 2024-04-15T22:23:21.981 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:23:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:23:21.888Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-15T22:23:22.838 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.176:9095"}]: dispatch 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.176:9095"}]: dispatch 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:22.839 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:23:22.840 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:22 smithi176 ceph-mon[25776]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:22.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:22.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:23:22.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:23:22.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.176:9095"}]: dispatch 2024-04-15T22:23:22.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.176:9095"}]: dispatch 2024-04-15T22:23:22.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:22.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:22.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:22.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:22.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:22.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:23:22.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:22.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:22.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:22.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:22.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:23:22.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[29727]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:22.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:22.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:23:22.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.176:9095"}]: dispatch 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.176:9095"}]: dispatch 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:23:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:22 smithi092 ceph-mon[24813]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:24.714 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:24 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:23:24] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:23:24.840 INFO:teuthology.orchestra.run.smithi092.stdout:133143986203 2024-04-15T22:23:24.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-15T22:23:25.126 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:24 smithi092 ceph-mon[29727]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:25.126 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:24 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:25.126 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:24 smithi092 ceph-mon[24813]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:25.126 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:24 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:25.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:24 smithi176 ceph-mon[25776]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:25.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:24 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:25.463 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:25 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:23:25] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:23:26.574 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:26 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:26.574 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:26 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:26.574 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:26 smithi092 ceph-mon[29727]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:26.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:26 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:26.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:26 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:26.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:26 smithi092 ceph-mon[24813]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:26.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:26 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:26.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:26 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:26.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:26 smithi176 ceph-mon[25776]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:27.000 INFO:teuthology.orchestra.run.smithi092.stdout:167503724567 2024-04-15T22:23:27.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-15T22:23:27.503 INFO:teuthology.orchestra.run.smithi092.stdout:34359738414 2024-04-15T22:23:27.503 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-15T22:23:28.473 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:28 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:23:28] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:23:29.142 INFO:teuthology.orchestra.run.smithi092.stdout:77309411366 2024-04-15T22:23:29.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-15T22:23:29.286 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:28 smithi092 ceph-mon[24813]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:29.286 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:28 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:29.286 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:28 smithi092 ceph-mon[29727]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:29.287 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:28 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:29.431 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:28 smithi176 ceph-mon[25776]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:29.431 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:28 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:29.600 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:29.601 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:29.625 INFO:teuthology.orchestra.run.smithi092.stdout:55834574890 2024-04-15T22:23:29.626 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-15T22:23:30.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:30 smithi092 ceph-mon[29727]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:30.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:30 smithi092 ceph-mon[24813]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:30.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:30 smithi176 ceph-mon[25776]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:30.734 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:30 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:23:30] "GET /metrics HTTP/1.1" 200 191485 "" "Prometheus/2.33.4" 2024-04-15T22:23:30.740 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:33.059 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:32 smithi092 ceph-mon[24813]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:33.059 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:32 smithi092 ceph-mon[29727]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:33.275 INFO:teuthology.orchestra.run.smithi092.stdout:201863462932 2024-04-15T22:23:33.275 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-15T22:23:33.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:32 smithi176 ceph-mon[25776]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:34.166 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:33 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:23:33] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:23:34.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:23:33 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:23:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:23:35.097 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:34 smithi092 ceph-mon[24813]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:35.097 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:34 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:35.097 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:34 smithi092 ceph-mon[29727]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:35.097 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:34 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:35.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:34 smithi176 ceph-mon[25776]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:35.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:34 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:36.096 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:36.096 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:36.475 INFO:teuthology.orchestra.run.smithi092.stdout:236223201295 2024-04-15T22:23:36.476 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-15T22:23:37.129 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:36 smithi092 ceph-mon[24813]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:37.129 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:36 smithi092 ceph-mon[29727]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:37.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:36 smithi176 ceph-mon[25776]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:38.055 INFO:teuthology.orchestra.run.smithi092.stdout:34359738418 2024-04-15T22:23:39.286 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:38 smithi176 ceph-mon[25776]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:39.286 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:38 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3325286716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T22:23:39.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:38 smithi092 ceph-mon[24813]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:39.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:38 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3325286716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T22:23:39.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:38 smithi092 ceph-mon[29727]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:39.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:38 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3325286716' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T22:23:40.186 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:40 smithi092 ceph-mon[24813]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:40.186 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:40 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2513688445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T22:23:40.186 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:40 smithi092 ceph-mon[29727]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:40.186 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:40 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2513688445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T22:23:40.387 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:40 smithi176 ceph-mon[25776]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:40.387 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:40 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2513688445' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T22:23:40.606 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738414 got 34359738418 for osd.0 2024-04-15T22:23:40.606 DEBUG:teuthology.parallel:result is None 2024-04-15T22:23:40.691 INFO:teuthology.orchestra.run.smithi092.stdout:167503724570 2024-04-15T22:23:40.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:23:40] "GET /metrics HTTP/1.1" 200 191484 "" "Prometheus/2.33.4" 2024-04-15T22:23:41.052 INFO:teuthology.orchestra.run.smithi092.stdout:133143986207 2024-04-15T22:23:41.313 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:41 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/1508447514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T22:23:41.313 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:41 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:41.313 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:41 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:41.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:41 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/1508447514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T22:23:41.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:41 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:41.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:41 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:41.314 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:41 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:23:41] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:23:41.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:41 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/1508447514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T22:23:41.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:41 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:41.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:41 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:42.374 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:42.374 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:42.384 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:42 smithi092 ceph-mon[29727]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:42.384 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:42 smithi092 ceph-mon[24813]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:42.391 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:42.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:42 smithi176 ceph-mon[25776]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:42.791 INFO:teuthology.orchestra.run.smithi092.stdout:77309411368 2024-04-15T22:23:43.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:43 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/477046974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T22:23:43.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:43 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/477046974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T22:23:43.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:43 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/477046974' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T22:23:43.520 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724567 got 167503724570 for osd.5 2024-04-15T22:23:43.520 DEBUG:teuthology.parallel:result is None 2024-04-15T22:23:44.155 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:44 smithi092 ceph-mon[24813]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:44.156 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:23:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:23:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:23:44.423 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:44 smithi092 ceph-mon[29727]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:44.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:44 smithi176 ceph-mon[25776]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:44.826 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986203 got 133143986207 for osd.4 2024-04-15T22:23:44.827 DEBUG:teuthology.parallel:result is None 2024-04-15T22:23:45.262 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411366 got 77309411368 for osd.2 2024-04-15T22:23:45.262 DEBUG:teuthology.parallel:result is None 2024-04-15T22:23:45.954 INFO:teuthology.orchestra.run.smithi092.stdout:111669149733 2024-04-15T22:23:46.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:46 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/4119828153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T22:23:46.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:23:46.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:23:46.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:46 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/4119828153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T22:23:46.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:23:46.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:23:46.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:46 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/4119828153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T22:23:46.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:23:46.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:23:47.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:47 smithi092 ceph-mon[29727]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:47.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:47 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:23:47.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:23:47.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:47 smithi092 ceph-mon[24813]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:47.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:47 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:23:47.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:23:47.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:47 smithi176 ceph-mon[25776]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:47.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:47 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:23:47.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:23:47.730 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149727 got 111669149733 for osd.3 2024-04-15T22:23:47.731 DEBUG:teuthology.parallel:result is None 2024-04-15T22:23:48.482 INFO:teuthology.orchestra.run.smithi092.stdout:236223201299 2024-04-15T22:23:49.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:49 smithi092 ceph-mon[24813]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:49.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:49 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2934021020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T22:23:49.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:49 smithi092 ceph-mon[29727]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:49.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:49 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2934021020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T22:23:49.436 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201295 got 236223201299 for osd.7 2024-04-15T22:23:49.436 DEBUG:teuthology.parallel:result is None 2024-04-15T22:23:49.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:49 smithi176 ceph-mon[25776]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:49.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:49 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2934021020' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T22:23:49.875 INFO:teuthology.orchestra.run.smithi092.stdout:55834574895 2024-04-15T22:23:50.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:50 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/285779512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T22:23:50.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:50 smithi092 ceph-mon[29727]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:50.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:50 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/285779512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T22:23:50.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:50 smithi092 ceph-mon[24813]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:50.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:50 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/285779512' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T22:23:50.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:50 smithi176 ceph-mon[25776]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:50.686 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:23:50] "GET /metrics HTTP/1.1" 200 191485 "" "Prometheus/2.33.4" 2024-04-15T22:23:51.348 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574890 got 55834574895 for osd.1 2024-04-15T22:23:51.348 DEBUG:teuthology.parallel:result is None 2024-04-15T22:23:52.456 INFO:teuthology.orchestra.run.smithi092.stdout:201863462935 2024-04-15T22:23:53.032 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:52 smithi092 ceph-mon[29727]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:53.032 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:52 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2737041740' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T22:23:53.032 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:52 smithi092 ceph-mon[24813]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:53.032 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:52 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2737041740' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T22:23:53.033 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462932 got 201863462935 for osd.6 2024-04-15T22:23:53.034 DEBUG:teuthology.parallel:result is None 2024-04-15T22:23:53.034 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T22:23:53.034 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T22:23:53.296 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:53.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:52 smithi176 ceph-mon[25776]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:53.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:52 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2737041740' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T22:23:54.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:23:53 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:23:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:23:54.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:54 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:23:54] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:23:55.137 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:54 smithi092 ceph-mon[24813]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:55.137 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:54 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:55.138 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:54 smithi092 ceph-mon[29727]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:55.138 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:54 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:55.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:54 smithi176 ceph-mon[25776]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:55.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:54 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:55.583 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:23:55.584 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-04-15T22:23:56.713 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:23:56] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:23:57.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[29727]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:23:57.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[29727]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:57.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:57.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:57.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:57.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:57.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:57.050 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:57.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[24813]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:23:57.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[24813]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:57.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:57.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:57.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:57.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:57.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:57.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:56 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:57.071 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-15T22:23:53.938683+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48968,"kb_used_data":6152,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912248,"statfs":{"total":767960285184,"available":767910141952,"internally_reserved":0,"allocated":6299648,"data_stored":3510888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002553"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-15T22:22:46.008543+0000","last_change":"2024-04-15T22:22:31.832054+0000","last_active":"2024-04-15T22:22:46.008543+0000","last_peered":"2024-04-15T22:22:46.008543+0000","last_clean":"2024-04-15T22:22:46.008543+0000","last_became_active":"2024-04-15T22:22:31.831619+0000","last_became_peered":"2024-04-15T22:22:31.831619+0000","last_unstale":"2024-04-15T22:22:46.008543+0000","last_undegraded":"2024-04-15T22:22:46.008543+0000","last_fullsized":"2024-04-15T22:22:46.008543+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T22:20:28.678152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T22:20:28.678152+0000","last_clean_scrub_stamp":"2024-04-15T22:20:28.678152+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T07:35:12.591740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201300,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":9.516}]},{"osd":1,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.375},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39600000000000002}]},{"osd":2,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.46300000000000002}]},{"osd":3,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.53900000000000003}]},{"osd":6,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.58099999999999996}]}]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.65300000000000002}]},{"osd":1,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.63600000000000001}]},{"osd":2,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.69499999999999995}]},{"osd":3,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.72199999999999998}]},{"osd":4,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.623},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.66600000000000004}]},{"osd":7,"last update":"Mon Apr 15 22:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.753},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.74399999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.33100000000000002,"15min":0.314},"min":{"1min":0.22500000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.34100000000000003,"15min":0.32100000000000001},"min":{"1min":0.251,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.40100000000000002}]},{"osd":2,"last update":"Mon Apr 15 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.375},"min":{"1min":0.27800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.377,"5min":0.37,"15min":0.36099999999999999},"min":{"1min":0.25900000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.26600000000000001}]},{"osd":3,"last update":"Mon Apr 15 22:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.40799999999999997,"15min":0.40300000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.442,"15min":0.439},"min":{"1min":0.33700000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"last":0.371}]},{"osd":4,"last update":"Mon Apr 15 22:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.57899999999999996},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.75600000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.58799999999999997}]},{"osd":5,"last update":"Mon Apr 15 22:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.64300000000000002,"15min":0.64900000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.64200000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.77500000000000002}]},{"osd":6,"last update":"Mon Apr 15 22:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.749}]},{"osd":7,"last update":"Mon Apr 15 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.625},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.65800000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 15 22:23:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.34499999999999997,"15min":0.32900000000000001},"min":{"1min":0.253,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.59299999999999997,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.36099999999999999,"15min":0.33900000000000002},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.65500000000000003,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.34200000000000003}]},{"osd":2,"last update":"Mon Apr 15 22:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35299999999999998,"15min":0.34100000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.379,"15min":0.36299999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64300000000000002,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.35199999999999998}]},{"osd":3,"last update":"Mon Apr 15 22:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.438,"15min":0.435},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.58199999999999996,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.45900000000000002,"15min":0.45500000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.68600000000000005,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.40600000000000003}]},{"osd":4,"last update":"Mon Apr 15 22:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.38100000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.39500000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73199999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.496}]},{"osd":5,"last update":"Mon Apr 15 22:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.63500000000000001,"15min":0.64800000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.73899999999999999,"5min":0.876,"15min":0.876},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64700000000000002,"15min":0.65700000000000003},"min":{"1min":0.437,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86699999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.64700000000000002}]},{"osd":6,"last update":"Mon Apr 15 22:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.61399999999999999}]},{"osd":7,"last update":"Mon Apr 15 22:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59299999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.34699999999999998,"15min":0.34300000000000003},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35499999999999998,"15min":0.34999999999999998},"min":{"1min":0.255,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54100000000000004,"5min":0.622,"15min":0.622},"last":0.27900000000000003}]},{"osd":1,"last update":"Mon Apr 15 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.371,"15min":0.36899999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.50900000000000001,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.435},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.373,"15min":0.36899999999999999},"min":{"1min":0.27600000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.35099999999999998}]},{"osd":3,"last update":"Mon Apr 15 22:22:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.41599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.58699999999999997,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28399999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.375}]},{"osd":4,"last update":"Mon Apr 15 22:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.38500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.72199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.51800000000000002}]},{"osd":5,"last update":"Mon Apr 15 22:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.58699999999999997,"15min":0.59699999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76600000000000001,"5min":1.212,"15min":1.212},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Apr 15 22:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.62}]},{"osd":7,"last update":"Mon Apr 15 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.629}]}]},{"osd":3,"up_from":26,"seq":111669149735,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.38500000000000001,"15min":0.39100000000000001},"min":{"1min":0.254,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.318},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.39000000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.59599999999999997,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.38600000000000001}]},{"osd":1,"last update":"Mon Apr 15 22:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.40400000000000003,"15min":0.41099999999999998},"min":{"1min":0.27300000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61299999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.41399999999999998,"15min":0.41599999999999998},"min":{"1min":0.28199999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64800000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.32800000000000001}]},{"osd":2,"last update":"Mon Apr 15 22:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.309,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.372},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.435,"15min":0.436},"min":{"1min":0.31,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.36099999999999999}]},{"osd":4,"last update":"Mon Apr 15 22:23:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54000000000000004,"15min":0.53700000000000003},"min":{"1min":0.443,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.63100000000000001}]},{"osd":5,"last update":"Mon Apr 15 22:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58099999999999996,"15min":0.57999999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.63900000000000001}]},{"osd":6,"last update":"Mon Apr 15 22:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.71699999999999997}]},{"osd":7,"last update":"Mon Apr 15 22:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.75},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.62}]}]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52900000000000003,"15min":0.52700000000000002},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.44},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.505,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.46100000000000002}]},{"osd":1,"last update":"Mon Apr 15 22:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.498,"5min":0.5,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Apr 15 22:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.505,"15min":0.501},"min":{"1min":0.38100000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51600000000000001,"15min":0.51300000000000001},"min":{"1min":0.375,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.57899999999999996}]},{"osd":3,"last update":"Mon Apr 15 22:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52200000000000002,"15min":0.52100000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.48299999999999998}]},{"osd":5,"last update":"Mon Apr 15 22:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.40600000000000003,"15min":0.39700000000000002},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63700000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.39900000000000002,"15min":0.38300000000000001},"min":{"1min":0.35199999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.622}]},{"osd":6,"last update":"Mon Apr 15 22:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.58799999999999997}]},{"osd":7,"last update":"Mon Apr 15 22:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.51500000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":240072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.51500000000000001,"15min":0.52300000000000002},"min":{"1min":0.33700000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.58199999999999996,"5min":0.752,"15min":0.752},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.41099999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.628,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Mon Apr 15 22:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.626,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.65700000000000003,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.67400000000000004}]},{"osd":2,"last update":"Mon Apr 15 22:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.63700000000000001,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.754},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.61899999999999999,"5min":0.745,"15min":0.745},"last":0.70999999999999996}]},{"osd":3,"last update":"Mon Apr 15 22:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.64500000000000002,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.74099999999999999}]},{"osd":4,"last update":"Mon Apr 15 22:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.40400000000000003,"15min":0.39900000000000002},"min":{"1min":0.307,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58099999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.41099999999999998,"15min":0.40400000000000003},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.66100000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.82599999999999996}]},{"osd":6,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.80300000000000005}]},{"osd":7,"last update":"Mon Apr 15 22:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.81599999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T22:23:57.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T22:23:57.285 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:23:57.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:56 smithi176 ceph-mon[25776]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:23:57.482 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:56 smithi176 ceph-mon[25776]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:57.482 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:56 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:57.482 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:56 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:57.483 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:56 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:23:57.483 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:56 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:23:57.484 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:56 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:23:57.484 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:56 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:23:58.332 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:58 smithi176 ceph-mon[25776]: Deploying daemon alertmanager.a on smithi092 2024-04-15T22:23:58.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:58 smithi092 ceph-mon[29727]: Deploying daemon alertmanager.a on smithi092 2024-04-15T22:23:58.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:58 smithi092 ceph-mon[24813]: Deploying daemon alertmanager.a on smithi092 2024-04-15T22:23:59.248 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:23:59 smithi092 ceph-mon[24813]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:59.248 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:23:59 smithi092 ceph-mon[29727]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:23:59.404 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:23:59.405 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-04-15T22:23:59.450 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:23:59 smithi176 ceph-mon[25776]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:00.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:00 smithi092 ceph-mon[24813]: from='client.14613 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:24:00.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:00 smithi092 ceph-mon[24813]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:00.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:00 smithi092 ceph-mon[29727]: from='client.14613 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:24:00.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:00 smithi092 ceph-mon[29727]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:00.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:00 smithi176 ceph-mon[25776]: from='client.14613 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T22:24:00.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:00 smithi176 ceph-mon[25776]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:00.533 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-15T22:23:57.939427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48968,"kb_used_data":6152,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912248,"statfs":{"total":767960285184,"available":767910141952,"internally_reserved":0,"allocated":6299648,"data_stored":3510888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002454"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-15T22:22:46.008543+0000","last_change":"2024-04-15T22:22:31.832054+0000","last_active":"2024-04-15T22:22:46.008543+0000","last_peered":"2024-04-15T22:22:46.008543+0000","last_clean":"2024-04-15T22:22:46.008543+0000","last_became_active":"2024-04-15T22:22:31.831619+0000","last_became_peered":"2024-04-15T22:22:31.831619+0000","last_unstale":"2024-04-15T22:22:46.008543+0000","last_undegraded":"2024-04-15T22:22:46.008543+0000","last_fullsized":"2024-04-15T22:22:46.008543+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T22:20:28.678152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T22:20:28.678152+0000","last_clean_scrub_stamp":"2024-04-15T22:20:28.678152+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T07:35:12.591740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201301,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6096,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739056,"statfs":{"total":95995035648,"available":95988793344,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.58999999999999997}]},{"osd":1,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.53700000000000003}]},{"osd":2,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.58499999999999996}]},{"osd":3,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.502},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.442}]},{"osd":4,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.628}]},{"osd":5,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Apr 15 22:23:32 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.60399999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462937,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.59399999999999997}]},{"osd":1,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.502},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.67500000000000004}]},{"osd":2,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.751},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.621}]},{"osd":3,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.626},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.73199999999999998}]},{"osd":4,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.44,"5min":0.44,"15min":0.44},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.66400000000000003}]},{"osd":5,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.55500000000000005}]},{"osd":7,"last update":"Mon Apr 15 22:23:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.69399999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.33100000000000002,"15min":0.314},"min":{"1min":0.22500000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.432},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.34100000000000003,"15min":0.32100000000000001},"min":{"1min":0.251,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.35199999999999998}]},{"osd":2,"last update":"Mon Apr 15 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39300000000000002,"15min":0.375},"min":{"1min":0.27800000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.377,"5min":0.37,"15min":0.36099999999999999},"min":{"1min":0.25900000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.41999999999999998}]},{"osd":3,"last update":"Mon Apr 15 22:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.623,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.44,"15min":0.438},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.45600000000000002}]},{"osd":4,"last update":"Mon Apr 15 22:23:15 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.56299999999999994,"15min":0.56699999999999995},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.57899999999999996},"min":{"1min":0.41199999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.75600000000000001,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.59399999999999997}]},{"osd":5,"last update":"Mon Apr 15 22:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.64300000000000002,"15min":0.64900000000000002},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.627},{"interface":"front","average":{"1min":0.623,"5min":0.63900000000000001,"15min":0.64200000000000002},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.60899999999999999}]},{"osd":6,"last update":"Mon Apr 15 22:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.55800000000000005}]},{"osd":7,"last update":"Mon Apr 15 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.61599999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 15 22:23:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.34499999999999997,"15min":0.32900000000000001},"min":{"1min":0.253,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.59299999999999997,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.31},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.36099999999999999,"15min":0.33900000000000002},"min":{"1min":0.25600000000000001,"5min":0.219,"15min":0.219},"max":{"1min":0.65500000000000003,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Apr 15 22:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35299999999999998,"15min":0.34100000000000003},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.379,"15min":0.36299999999999999},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.64300000000000002,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.26500000000000001}]},{"osd":3,"last update":"Mon Apr 15 22:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.42199999999999999,"15min":0.42999999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.46100000000000002,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.44600000000000001,"15min":0.45100000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.54700000000000004,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.38600000000000001}]},{"osd":4,"last update":"Mon Apr 15 22:23:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55400000000000005,"15min":0.55200000000000005},"min":{"1min":0.38100000000000001,"5min":0.318,"15min":0.318},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55100000000000005,"15min":0.54700000000000004},"min":{"1min":0.39500000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.73199999999999998,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.432}]},{"osd":5,"last update":"Mon Apr 15 22:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.63500000000000001,"15min":0.64800000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.73899999999999999,"5min":0.876,"15min":0.876},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64700000000000002,"15min":0.65700000000000003},"min":{"1min":0.437,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86699999999999999,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.60999999999999999}]},{"osd":6,"last update":"Mon Apr 15 22:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.57299999999999995}]},{"osd":7,"last update":"Mon Apr 15 22:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.56200000000000006}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.34699999999999998,"15min":0.34300000000000003},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.35499999999999998,"15min":0.34999999999999998},"min":{"1min":0.255,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54100000000000004,"5min":0.622,"15min":0.622},"last":0.27100000000000002}]},{"osd":1,"last update":"Mon Apr 15 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.371,"15min":0.36899999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.50900000000000001,"5min":0.53600000000000003,"15min":0.53600000000000003},"last":0.249},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.373,"15min":0.36899999999999999},"min":{"1min":0.27600000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59399999999999997,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.36899999999999999}]},{"osd":3,"last update":"Mon Apr 15 22:22:57 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.41599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.58699999999999997,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.28399999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.316}]},{"osd":4,"last update":"Mon Apr 15 22:23:18 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53400000000000003,"15min":0.53600000000000003},"min":{"1min":0.38500000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.72199999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.45300000000000001}]},{"osd":5,"last update":"Mon Apr 15 22:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.58699999999999997,"15min":0.59699999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76600000000000001,"5min":1.212,"15min":1.212},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57699999999999996,"15min":0.57899999999999996},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55400000000000005}]},{"osd":6,"last update":"Mon Apr 15 22:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.51800000000000002}]},{"osd":7,"last update":"Mon Apr 15 22:23:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.52500000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.38500000000000001,"15min":0.39100000000000001},"min":{"1min":0.254,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.39000000000000001,"15min":0.39100000000000001},"min":{"1min":0.25700000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.59599999999999997,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.35799999999999998}]},{"osd":1,"last update":"Mon Apr 15 22:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.40400000000000003,"15min":0.41099999999999998},"min":{"1min":0.27300000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61299999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.41399999999999998,"15min":0.41599999999999998},"min":{"1min":0.28199999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.64800000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.28399999999999997}]},{"osd":2,"last update":"Mon Apr 15 22:23:51 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.41499999999999998,"15min":0.41499999999999998},"min":{"1min":0.309,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.435,"15min":0.436},"min":{"1min":0.31,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.314}]},{"osd":4,"last update":"Mon Apr 15 22:23:16 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.41899999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.54000000000000004,"15min":0.53700000000000003},"min":{"1min":0.443,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.64800000000000002}]},{"osd":5,"last update":"Mon Apr 15 22:23:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.60099999999999998,"15min":0.60899999999999999},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58099999999999996,"15min":0.57999999999999996},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56799999999999995}]},{"osd":6,"last update":"Mon Apr 15 22:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.53400000000000003}]},{"osd":7,"last update":"Mon Apr 15 22:23:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.51100000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986211,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.52900000000000003,"15min":0.52700000000000002},"min":{"1min":0.33300000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.505,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.52200000000000002}]},{"osd":1,"last update":"Mon Apr 15 22:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.51800000000000002,"15min":0.51400000000000001},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.5,"15min":0.5},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.49399999999999999}]},{"osd":2,"last update":"Mon Apr 15 22:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.505,"15min":0.501},"min":{"1min":0.38100000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51600000000000001,"15min":0.51300000000000001},"min":{"1min":0.375,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Apr 15 22:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52200000000000002,"15min":0.52100000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.59399999999999997}]},{"osd":5,"last update":"Mon Apr 15 22:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.40600000000000003,"15min":0.39700000000000002},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.63700000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.39900000000000002,"15min":0.38300000000000001},"min":{"1min":0.35199999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.60999999999999999}]},{"osd":6,"last update":"Mon Apr 15 22:23:03 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.57099999999999995}]},{"osd":7,"last update":"Mon Apr 15 22:23:31 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.55600000000000005}]}]},{"osd":5,"up_from":39,"seq":167503724574,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":240072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 22:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.51500000000000001,"15min":0.52300000000000002},"min":{"1min":0.33700000000000002,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.58199999999999996,"5min":0.752,"15min":0.752},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.41099999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.628,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.52000000000000002}]},{"osd":1,"last update":"Mon Apr 15 22:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.54700000000000004,"15min":0.55400000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.626,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.65700000000000003,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.48499999999999999}]},{"osd":2,"last update":"Mon Apr 15 22:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.54500000000000004,"15min":0.55300000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.63700000000000001,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54700000000000004,"15min":0.55300000000000005},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.61899999999999999,"5min":0.745,"15min":0.745},"last":0.57899999999999996}]},{"osd":3,"last update":"Mon Apr 15 22:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.52800000000000002,"15min":0.53400000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.64500000000000002,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.57599999999999996,"15min":0.58199999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.61799999999999999}]},{"osd":4,"last update":"Mon Apr 15 22:23:39 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.40400000000000003,"15min":0.39900000000000002},"min":{"1min":0.307,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58099999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.41099999999999998,"15min":0.40400000000000003},"min":{"1min":0.34399999999999997,"5min":0.252,"15min":0.252},"max":{"1min":0.66100000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.503}]},{"osd":6,"last update":"Mon Apr 15 22:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.55400000000000005}]},{"osd":7,"last update":"Mon Apr 15 22:23:30 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.60899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T22:24:00.534 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T22:24:00.534 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-15T22:24:00.534 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-15T22:24:00.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -- ceph health --format=json 2024-04-15T22:24:00.897 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T22:24:00.963 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:24:00] "GET /metrics HTTP/1.1" 200 191485 "" "Prometheus/2.33.4" 2024-04-15T22:24:01.714 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:01 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:24:01] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:24:02.450 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:02 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:02.450 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:02 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:02.450 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:02 smithi092 ceph-mon[29727]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:02.451 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:02 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:02.451 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:02 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:02.451 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:02 smithi092 ceph-mon[24813]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:02.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:02 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:02.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:02 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:02.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:02 smithi176 ceph-mon[25776]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:04.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:24:03 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:24:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:24:05.024 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:04 smithi092 ceph-mon[24813]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:05.025 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:04 smithi092 ceph-mon[29727]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:05.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:04 smithi176 ceph-mon[25776]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:05.834 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-04-15T22:24:05.834 INFO:teuthology.orchestra.run.smithi092.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-15T22:24:06.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:05 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3718567206' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-15T22:24:06.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:05 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3718567206' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-15T22:24:06.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:05 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3718567206' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-15T22:24:06.702 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-15T22:24:06.702 INFO:tasks.cephadm:Setup complete, yielding 2024-04-15T22:24:06.703 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T22:24:06.713 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi092.front.sepia.ceph.com 2024-04-15T22:24:06.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-04-15T22:24:07.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:07 smithi092 ceph-mon[24813]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:07.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:07 smithi092 ceph-mon[29727]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:07.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:07 smithi176 ceph-mon[25776]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:08.593 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:08 smithi092 podman[49957]: 2024-04-15 22:24:08.47628272 +0000 UTC m=+1.760248219 volume create 130b5b58c998f0632cf04f8338487adba00dbe04b759bffd43d6d10d94f56064 2024-04-15T22:24:08.593 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:08 smithi092 podman[49957]: 2024-04-15 22:24:08.59357989 +0000 UTC m=+1.877545388 container create b3f677ebd5acad0fe6c37c62d45ece64ed51a501b363f05ccbda400ba3587b92 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:09.171 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:09 smithi092 ceph-mon[24813]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:09.171 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:09 smithi092 ceph-mon[29727]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:09.464 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:09 smithi092 podman[49957]: 2024-04-15 22:24:09.24807232 +0000 UTC m=+2.532037820 container init b3f677ebd5acad0fe6c37c62d45ece64ed51a501b363f05ccbda400ba3587b92 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:09.464 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:09 smithi092 podman[49957]: 2024-04-15 22:24:09.25081477 +0000 UTC m=+2.534780277 container start b3f677ebd5acad0fe6c37c62d45ece64ed51a501b363f05ccbda400ba3587b92 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:09.464 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:09 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[50075]: level=info ts=2024-04-15T22:24:09.260Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-15T22:24:09.464 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:09 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[50075]: level=info ts=2024-04-15T22:24:09.260Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-15T22:24:09.464 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:09 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[50075]: level=info ts=2024-04-15T22:24:09.261Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.92 port=9094 2024-04-15T22:24:09.465 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:09 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[50075]: level=info ts=2024-04-15T22:24:09.262Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-15T22:24:09.465 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:09 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[50075]: level=info ts=2024-04-15T22:24:09.287Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-15T22:24:09.465 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:09 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[50075]: level=info ts=2024-04-15T22:24:09.287Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-15T22:24:09.465 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:09 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[50075]: level=info ts=2024-04-15T22:24:09.289Z caller=main.go:518 msg=Listening address=:9093 2024-04-15T22:24:09.465 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:09 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[50075]: level=info ts=2024-04-15T22:24:09.289Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-15T22:24:09.465 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:09 smithi092 bash[49957]: b3f677ebd5acad0fe6c37c62d45ece64ed51a501b363f05ccbda400ba3587b92 2024-04-15T22:24:09.465 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:09 smithi092 systemd[1]: Started Ceph alertmanager.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:24:09.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:09 smithi176 ceph-mon[25776]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:10.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:10.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:10.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:10.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T22:24:10.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T22:24:10.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:10.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[24813]: Deploying daemon grafana.a on smithi176 2024-04-15T22:24:10.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[24813]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:10.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:10.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:10.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:10.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T22:24:10.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[29727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T22:24:10.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:10.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[29727]: Deploying daemon grafana.a on smithi176 2024-04-15T22:24:10.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-mon[29727]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:10.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:10 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:10.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:10 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:10.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:10 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:10.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:10 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T22:24:10.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:10 smithi176 ceph-mon[25776]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T22:24:10.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:10 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:10.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:10 smithi176 ceph-mon[25776]: Deploying daemon grafana.a on smithi176 2024-04-15T22:24:10.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:10 smithi176 ceph-mon[25776]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:10.865 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:10 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:24:10] "GET /metrics HTTP/1.1" 200 191488 "" "Prometheus/2.33.4" 2024-04-15T22:24:11.464 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:11 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[50075]: level=info ts=2024-04-15T22:24:11.262Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000159873s 2024-04-15T22:24:12.055 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:12 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:12.055 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:12 smithi176 ceph-mon[25776]: osdmap e61: 8 total, 8 up, 8 in 2024-04-15T22:24:12.055 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:12 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/1713507843' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-15T22:24:12.055 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:12 smithi176 ceph-mon[25776]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-15T22:24:12.055 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:12 smithi176 ceph-mon[25776]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:12.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:12 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:12.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:12 smithi092 ceph-mon[29727]: osdmap e61: 8 total, 8 up, 8 in 2024-04-15T22:24:12.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:12 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/1713507843' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-15T22:24:12.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:12 smithi092 ceph-mon[29727]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-15T22:24:12.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:12 smithi092 ceph-mon[29727]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:12.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:12 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:12.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:12 smithi092 ceph-mon[24813]: osdmap e61: 8 total, 8 up, 8 in 2024-04-15T22:24:12.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:12 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/1713507843' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-15T22:24:12.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:12 smithi092 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-15T22:24:12.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:12 smithi092 ceph-mon[24813]: pgmap v47: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:12.521 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-15T22:24:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "335209a9-9b58-4290-9489-e15dd7ac5528", 2024-04-15T22:24:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "r", 2024-04-15T22:24:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: "current_period": "ee746efb-c2e1-498c-a013-2f5f750a6b5d", 2024-04-15T22:24:12.522 INFO:teuthology.orchestra.run.smithi092.stdout: "epoch": 1 2024-04-15T22:24:12.522 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-15T22:24:13.373 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-04-15T22:24:13.600 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:13 smithi092 ceph-mon[29727]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-15T22:24:13.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:13 smithi092 ceph-mon[29727]: osdmap e62: 8 total, 8 up, 8 in 2024-04-15T22:24:13.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:13 smithi092 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-15T22:24:13.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:13 smithi092 ceph-mon[24813]: osdmap e62: 8 total, 8 up, 8 in 2024-04-15T22:24:13.778 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:13 smithi176 ceph-mon[25776]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-15T22:24:13.779 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:13 smithi176 ceph-mon[25776]: osdmap e62: 8 total, 8 up, 8 in 2024-04-15T22:24:14.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:24:13 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:24:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:24:14.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:14 smithi176 ceph-mon[25776]: osdmap e63: 8 total, 8 up, 8 in 2024-04-15T22:24:14.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:14 smithi176 ceph-mon[25776]: pgmap v50: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:14.793 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:14 smithi092 ceph-mon[29727]: osdmap e63: 8 total, 8 up, 8 in 2024-04-15T22:24:14.793 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:14 smithi092 ceph-mon[29727]: pgmap v50: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:14.794 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:14 smithi092 ceph-mon[24813]: osdmap e63: 8 total, 8 up, 8 in 2024-04-15T22:24:14.794 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:14 smithi092 ceph-mon[24813]: pgmap v50: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:16.387 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:16 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:24:16] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:24:17.336 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:16 smithi176 ceph-mon[25776]: pgmap v51: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:17.337 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:16 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:17.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:16 smithi092 ceph-mon[29727]: pgmap v51: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:17.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:16 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:17.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:16 smithi092 ceph-mon[24813]: pgmap v51: 33 pgs: 32 creating+peering, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:17.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:16 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:17.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:17 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:24:17] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:24:18.198 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:18 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:18.198 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:18 smithi092 ceph-mon[24813]: pgmap v52: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-15T22:24:18.199 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:18 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:18.199 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:18 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:18.199 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:18 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:18.199 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:18 smithi092 ceph-mon[29727]: pgmap v52: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-15T22:24:18.199 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:18 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:18.199 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:18 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:18.199 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:18 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:24:18] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:24:18.253 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-15T22:24:18.253 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "861582b5-4be2-42e3-87c8-68fd8cc06060", 2024-04-15T22:24:18.253 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "default", 2024-04-15T22:24:18.253 INFO:teuthology.orchestra.run.smithi092.stdout: "api_name": "default", 2024-04-15T22:24:18.253 INFO:teuthology.orchestra.run.smithi092.stdout: "is_master": "true", 2024-04-15T22:24:18.254 INFO:teuthology.orchestra.run.smithi092.stdout: "endpoints": [], 2024-04-15T22:24:18.254 INFO:teuthology.orchestra.run.smithi092.stdout: "hostnames": [], 2024-04-15T22:24:18.254 INFO:teuthology.orchestra.run.smithi092.stdout: "hostnames_s3website": [], 2024-04-15T22:24:18.254 INFO:teuthology.orchestra.run.smithi092.stdout: "master_zone": "", 2024-04-15T22:24:18.254 INFO:teuthology.orchestra.run.smithi092.stdout: "zones": [], 2024-04-15T22:24:18.254 INFO:teuthology.orchestra.run.smithi092.stdout: "placement_targets": [], 2024-04-15T22:24:18.254 INFO:teuthology.orchestra.run.smithi092.stdout: "default_placement": "", 2024-04-15T22:24:18.254 INFO:teuthology.orchestra.run.smithi092.stdout: "realm_id": "335209a9-9b58-4290-9489-e15dd7ac5528", 2024-04-15T22:24:18.254 INFO:teuthology.orchestra.run.smithi092.stdout: "sync_policy": { 2024-04-15T22:24:18.254 INFO:teuthology.orchestra.run.smithi092.stdout: "groups": [] 2024-04-15T22:24:18.254 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-15T22:24:18.254 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-15T22:24:18.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:18 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:18.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:18 smithi176 ceph-mon[25776]: pgmap v52: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 1 op/s 2024-04-15T22:24:18.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:18 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:18.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:18 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:18.933 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-04-15T22:24:19.551 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:19 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[50075]: level=info ts=2024-04-15T22:24:19.263Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001485637s 2024-04-15T22:24:20.171 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:20 smithi092 ceph-mon[24813]: pgmap v53: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 1 op/s 2024-04-15T22:24:20.171 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:20 smithi092 ceph-mon[29727]: pgmap v53: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 1 op/s 2024-04-15T22:24:20.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:20 smithi176 ceph-mon[25776]: pgmap v53: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 1 op/s 2024-04-15T22:24:20.632 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-15T22:24:20.633 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "94ef375c-fc99-4597-867f-a8070954ca57", 2024-04-15T22:24:20.633 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "z", 2024-04-15T22:24:20.633 INFO:teuthology.orchestra.run.smithi092.stdout: "domain_root": "z.rgw.meta:root", 2024-04-15T22:24:20.633 INFO:teuthology.orchestra.run.smithi092.stdout: "control_pool": "z.rgw.control", 2024-04-15T22:24:20.633 INFO:teuthology.orchestra.run.smithi092.stdout: "gc_pool": "z.rgw.log:gc", 2024-04-15T22:24:20.633 INFO:teuthology.orchestra.run.smithi092.stdout: "lc_pool": "z.rgw.log:lc", 2024-04-15T22:24:20.633 INFO:teuthology.orchestra.run.smithi092.stdout: "log_pool": "z.rgw.log", 2024-04-15T22:24:20.633 INFO:teuthology.orchestra.run.smithi092.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-04-15T22:24:20.633 INFO:teuthology.orchestra.run.smithi092.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-04-15T22:24:20.634 INFO:teuthology.orchestra.run.smithi092.stdout: "roles_pool": "z.rgw.meta:roles", 2024-04-15T22:24:20.634 INFO:teuthology.orchestra.run.smithi092.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-04-15T22:24:20.634 INFO:teuthology.orchestra.run.smithi092.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-04-15T22:24:20.634 INFO:teuthology.orchestra.run.smithi092.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-04-15T22:24:20.634 INFO:teuthology.orchestra.run.smithi092.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-04-15T22:24:20.634 INFO:teuthology.orchestra.run.smithi092.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-04-15T22:24:20.634 INFO:teuthology.orchestra.run.smithi092.stdout: "otp_pool": "z.rgw.otp", 2024-04-15T22:24:20.634 INFO:teuthology.orchestra.run.smithi092.stdout: "system_key": { 2024-04-15T22:24:20.635 INFO:teuthology.orchestra.run.smithi092.stdout: "access_key": "", 2024-04-15T22:24:20.635 INFO:teuthology.orchestra.run.smithi092.stdout: "secret_key": "" 2024-04-15T22:24:20.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:24:20.635 INFO:teuthology.orchestra.run.smithi092.stdout: "placement_pools": [ 2024-04-15T22:24:20.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-15T22:24:20.635 INFO:teuthology.orchestra.run.smithi092.stdout: "key": "default-placement", 2024-04-15T22:24:20.635 INFO:teuthology.orchestra.run.smithi092.stdout: "val": { 2024-04-15T22:24:20.635 INFO:teuthology.orchestra.run.smithi092.stdout: "index_pool": "z.rgw.buckets.index", 2024-04-15T22:24:20.636 INFO:teuthology.orchestra.run.smithi092.stdout: "storage_classes": { 2024-04-15T22:24:20.636 INFO:teuthology.orchestra.run.smithi092.stdout: "STANDARD": { 2024-04-15T22:24:20.636 INFO:teuthology.orchestra.run.smithi092.stdout: "data_pool": "z.rgw.buckets.data" 2024-04-15T22:24:20.636 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-15T22:24:20.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:24:20.636 INFO:teuthology.orchestra.run.smithi092.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-04-15T22:24:20.636 INFO:teuthology.orchestra.run.smithi092.stdout: "index_type": 0 2024-04-15T22:24:20.636 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-15T22:24:20.636 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-15T22:24:20.636 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-04-15T22:24:20.637 INFO:teuthology.orchestra.run.smithi092.stdout: "realm_id": "335209a9-9b58-4290-9489-e15dd7ac5528", 2024-04-15T22:24:20.637 INFO:teuthology.orchestra.run.smithi092.stdout: "notif_pool": "z.rgw.log:notif" 2024-04-15T22:24:20.637 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-15T22:24:20.713 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:20 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:24:20] "GET /metrics HTTP/1.1" 200 192605 "" "Prometheus/2.33.4" 2024-04-15T22:24:21.228 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-04-15T22:24:21.463 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:21 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:24:21] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:24:22.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:22 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:22.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:22 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:22.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:22 smithi176 ceph-mon[25776]: pgmap v54: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 866 B/s wr, 1 op/s 2024-04-15T22:24:22.628 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:22 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:22.628 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:22 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:22.628 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:22 smithi092 ceph-mon[29727]: pgmap v54: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 866 B/s wr, 1 op/s 2024-04-15T22:24:22.628 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:22 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:22.628 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:22 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:22.628 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:22 smithi092 ceph-mon[24813]: pgmap v54: 33 pgs: 33 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 866 B/s wr, 1 op/s 2024-04-15T22:24:23.978 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:24:23 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:24:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:24:24.916 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:24 smithi176 ceph-mon[25776]: osdmap e64: 8 total, 8 up, 8 in 2024-04-15T22:24:24.916 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:24 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/126385837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-15T22:24:24.917 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:24 smithi176 ceph-mon[25776]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-15T22:24:24.917 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:24 smithi176 ceph-mon[25776]: pgmap v56: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-15T22:24:24.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:24 smithi092 ceph-mon[24813]: osdmap e64: 8 total, 8 up, 8 in 2024-04-15T22:24:24.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:24 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/126385837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-15T22:24:24.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:24 smithi092 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-15T22:24:24.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:24 smithi092 ceph-mon[24813]: pgmap v56: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-15T22:24:24.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:24 smithi092 ceph-mon[29727]: osdmap e64: 8 total, 8 up, 8 in 2024-04-15T22:24:24.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:24 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/126385837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-15T22:24:24.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:24 smithi092 ceph-mon[29727]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-15T22:24:24.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:24 smithi092 ceph-mon[29727]: pgmap v56: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-15T22:24:25.478 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 systemd[1]: Starting Ceph grafana.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:24:25.796 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-mon[25776]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-15T22:24:25.796 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-mon[25776]: osdmap e65: 8 total, 8 up, 8 in 2024-04-15T22:24:25.797 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 podman[44638]: 2024-04-15 22:24:25.434061157 +0000 UTC m=+0.017299163 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-15T22:24:25.797 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 podman[44638]: 2024-04-15 22:24:25.560358379 +0000 UTC m=+0.143596412 container create 1233e94881e9603abe96e431fd04326e6163d4488f47816024b9ba4ebd7f7389 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , distribution-scope=public, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, io.buildah.version=1.24.2, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., version=8.5, name=ubi8) 2024-04-15T22:24:25.797 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 podman[44638]: 2024-04-15 22:24:25.797282817 +0000 UTC m=+0.380520815 container init 1233e94881e9603abe96e431fd04326e6163d4488f47816024b9ba4ebd7f7389 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=236.1648460182, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, build-date=2022-03-28T10:36:18.413762, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , io.buildah.version=1.24.2, version=8.5, architecture=x86_64, com.redhat.component=ubi8-container, io.openshift.expose-services=, io.openshift.tags=base rhel8, distribution-scope=public) 2024-04-15T22:24:25.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:25 smithi092 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-15T22:24:25.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:25 smithi092 ceph-mon[24813]: osdmap e65: 8 total, 8 up, 8 in 2024-04-15T22:24:25.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:25 smithi092 ceph-mon[29727]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-15T22:24:25.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:25 smithi092 ceph-mon[29727]: osdmap e65: 8 total, 8 up, 8 in 2024-04-15T22:24:26.049 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 podman[44638]: 2024-04-15 22:24:25.800931946 +0000 UTC m=+0.384169937 container start 1233e94881e9603abe96e431fd04326e6163d4488f47816024b9ba4ebd7f7389 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a, release=236.1648460182, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-type=git, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., version=8.5, com.redhat.component=ubi8-container, io.openshift.tags=base rhel8, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, io.buildah.version=1.24.2, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-15T22:24:26.049 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-04-15T22:24:26.049 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-04-15T22:24:26.049 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-04-15T22:24:26.049 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-04-15T22:24:26.049 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-15T22:24:26.049 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-15T22:24:26.050 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-15T22:24:26.050 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-15T22:24:26.050 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-04-15T22:24:26.050 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-04-15T22:24:26.050 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-04-15T22:24:26.050 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-04-15T22:24:26.052 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-04-15T22:24:26.052 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="App mode production" logger=settings 2024-04-15T22:24:26.052 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-04-15T22:24:26.052 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-15T22:24:26.052 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-04-15T22:24:26.052 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-04-15T22:24:26.053 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-04-15T22:24:26.053 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-04-15T22:24:26.053 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-04-15T22:24:26.053 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-04-15T22:24:26.053 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-04-15T22:24:26.053 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-04-15T22:24:26.053 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-04-15T22:24:26.053 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-04-15T22:24:26.053 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-04-15T22:24:26.053 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-15T22:24:26.053 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-04-15T22:24:26.054 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-04-15T22:24:26.054 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-04-15T22:24:26.054 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-04-15T22:24:26.054 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-04-15T22:24:26.054 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-04-15T22:24:26.054 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-04-15T22:24:26.054 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-04-15T22:24:26.054 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-04-15T22:24:26.054 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-04-15T22:24:26.054 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-04-15T22:24:26.054 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-04-15T22:24:26.055 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-04-15T22:24:26.055 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-04-15T22:24:26.055 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-04-15T22:24:26.055 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-04-15T22:24:26.055 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-04-15T22:24:26.055 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-04-15T22:24:26.055 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-15T22:24:26.055 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-04-15T22:24:26.055 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-04-15T22:24:26.055 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-04-15T22:24:26.055 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-04-15T22:24:26.055 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-04-15T22:24:26.056 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-04-15T22:24:26.056 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-04-15T22:24:26.056 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-04-15T22:24:26.056 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-04-15T22:24:26.056 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-04-15T22:24:26.056 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-04-15T22:24:26.056 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-04-15T22:24:26.056 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-04-15T22:24:26.056 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-04-15T22:24:26.056 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-04-15T22:24:26.056 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-04-15T22:24:26.057 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-04-15T22:24:26.057 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-04-15T22:24:26.057 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-04-15T22:24:26.057 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-04-15T22:24:26.057 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-04-15T22:24:26.057 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-04-15T22:24:26.057 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-04-15T22:24:26.057 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-04-15T22:24:26.057 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-15T22:24:26.057 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-04-15T22:24:26.057 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-04-15T22:24:26.057 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-04-15T22:24:26.058 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-04-15T22:24:26.058 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-04-15T22:24:26.058 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-04-15T22:24:26.058 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-04-15T22:24:26.058 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-04-15T22:24:26.058 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-04-15T22:24:26.058 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-04-15T22:24:26.058 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-04-15T22:24:26.058 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-04-15T22:24:26.058 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-04-15T22:24:26.058 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-04-15T22:24:26.058 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-04-15T22:24:26.059 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-04-15T22:24:26.059 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-04-15T22:24:26.059 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-04-15T22:24:26.059 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-04-15T22:24:26.059 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-04-15T22:24:26.059 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-04-15T22:24:26.059 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-04-15T22:24:26.059 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-04-15T22:24:26.059 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-04-15T22:24:26.059 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-15T22:24:26.059 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-04-15T22:24:26.060 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-15T22:24:26.060 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-04-15T22:24:26.060 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-15T22:24:26.060 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-15T22:24:26.060 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-04-15T22:24:26.060 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-04-15T22:24:26.060 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-04-15T22:24:26.060 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-04-15T22:24:26.060 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-04-15T22:24:26.060 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-04-15T22:24:26.060 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-04-15T22:24:26.060 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-04-15T22:24:26.061 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-04-15T22:24:26.061 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-04-15T22:24:26.061 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-04-15T22:24:26.061 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-04-15T22:24:26.061 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-04-15T22:24:26.061 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 bash[44638]: 1233e94881e9603abe96e431fd04326e6163d4488f47816024b9ba4ebd7f7389 2024-04-15T22:24:26.061 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 systemd[1]: Started Ceph grafana.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:24:26.061 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-04-15T22:24:26.061 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-04-15T22:24:26.061 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-04-15T22:24:26.061 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-15T22:24:26.061 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-04-15T22:24:26.061 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-04-15T22:24:26.062 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-04-15T22:24:26.062 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-04-15T22:24:26.062 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-04-15T22:24:26.062 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-04-15T22:24:26.062 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-04-15T22:24:26.062 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-04-15T22:24:26.062 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-04-15T22:24:26.062 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-04-15T22:24:26.062 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-04-15T22:24:26.062 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-04-15T22:24:26.062 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-04-15T22:24:26.063 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-04-15T22:24:26.063 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-04-15T22:24:26.063 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-04-15T22:24:26.063 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-04-15T22:24:26.063 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-04-15T22:24:26.063 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-04-15T22:24:26.063 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-04-15T22:24:26.063 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-04-15T22:24:26.063 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-04-15T22:24:26.063 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-04-15T22:24:26.063 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-04-15T22:24:26.063 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-04-15T22:24:26.063 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-04-15T22:24:26.064 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-04-15T22:24:26.064 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-04-15T22:24:26.064 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-04-15T22:24:26.064 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-04-15T22:24:26.064 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-04-15T22:24:26.064 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-04-15T22:24:26.064 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-04-15T22:24:26.065 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-15T22:24:26.065 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-15T22:24:26.065 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-04-15T22:24:26.065 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-04-15T22:24:26.065 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-04-15T22:24:26.065 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-04-15T22:24:26.065 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-15T22:24:26.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-04-15T22:24:26.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-15T22:24:26.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-04-15T22:24:26.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-04-15T22:24:26.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-15T22:24:26.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:25+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-04-15T22:24:26.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-04-15T22:24:26.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-04-15T22:24:26.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-04-15T22:24:26.066 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-04-15T22:24:26.067 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-04-15T22:24:26.067 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-04-15T22:24:26.067 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-04-15T22:24:26.067 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-04-15T22:24:26.067 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-04-15T22:24:26.067 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-04-15T22:24:26.067 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-04-15T22:24:26.067 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-04-15T22:24:26.067 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-04-15T22:24:26.067 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-04-15T22:24:26.067 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-04-15T22:24:26.067 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-04-15T22:24:26.068 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-04-15T22:24:26.068 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-04-15T22:24:26.068 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-04-15T22:24:26.068 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-04-15T22:24:26.068 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-15T22:24:26.068 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-15T22:24:26.068 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-15T22:24:26.068 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-04-15T22:24:26.068 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-15T22:24:26.068 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-04-15T22:24:26.068 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-04-15T22:24:26.068 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-04-15T22:24:26.069 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-04-15T22:24:26.069 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-04-15T22:24:26.069 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-04-15T22:24:26.069 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-04-15T22:24:26.069 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-04-15T22:24:26.069 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-04-15T22:24:26.069 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-04-15T22:24:26.069 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-04-15T22:24:26.069 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-15T22:24:26.069 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-04-15T22:24:26.069 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-04-15T22:24:26.069 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-15T22:24:26.069 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-04-15T22:24:26.070 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-04-15T22:24:26.070 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-04-15T22:24:26.070 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-04-15T22:24:26.301 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-04-15T22:24:26.302 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-04-15T22:24:26.302 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-04-15T22:24:26.302 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-04-15T22:24:26.302 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-04-15T22:24:26.302 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-04-15T22:24:26.302 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-04-15T22:24:26.302 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-04-15T22:24:26.302 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-04-15T22:24:26.303 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-04-15T22:24:26.303 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-04-15T22:24:26.303 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-04-15T22:24:26.303 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-04-15T22:24:26.303 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-04-15T22:24:26.303 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-04-15T22:24:26.303 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-04-15T22:24:26.303 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-04-15T22:24:26.303 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-15T22:24:26.303 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-15T22:24:26.305 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-15T22:24:26.305 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-04-15T22:24:26.305 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-15T22:24:26.305 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-04-15T22:24:26.305 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-04-15T22:24:26.305 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-04-15T22:24:26.305 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-04-15T22:24:26.305 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-04-15T22:24:26.306 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-04-15T22:24:26.306 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-04-15T22:24:26.306 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-04-15T22:24:26.306 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-04-15T22:24:26.306 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-04-15T22:24:26.306 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-04-15T22:24:26.306 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-04-15T22:24:26.306 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-04-15T22:24:26.306 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-15T22:24:26.307 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-15T22:24:26.307 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-15T22:24:26.307 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-15T22:24:26.307 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-04-15T22:24:26.307 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-04-15T22:24:26.307 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-04-15T22:24:26.307 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-04-15T22:24:26.307 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-15T22:24:26.307 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-04-15T22:24:26.307 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-04-15T22:24:26.307 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-04-15T22:24:26.308 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-04-15T22:24:26.308 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-04-15T22:24:26.308 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-04-15T22:24:26.308 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-04-15T22:24:26.308 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-04-15T22:24:26.308 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-04-15T22:24:26.308 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-04-15T22:24:26.308 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-04-15T22:24:26.308 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-04-15T22:24:26.308 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-04-15T22:24:26.308 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-04-15T22:24:26.309 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-04-15T22:24:26.309 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-15T22:24:26.309 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-15T22:24:26.309 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-04-15T22:24:26.309 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-04-15T22:24:26.309 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-04-15T22:24:26.309 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-04-15T22:24:26.309 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-04-15T22:24:26.309 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-04-15T22:24:26.309 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-04-15T22:24:26.309 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-04-15T22:24:26.310 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-04-15T22:24:26.310 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-04-15T22:24:26.310 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-04-15T22:24:26.310 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-15T22:24:26.310 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-04-15T22:24:26.310 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-04-15T22:24:26.310 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-04-15T22:24:26.310 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-04-15T22:24:26.310 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-04-15T22:24:26.310 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-15T22:24:26.310 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-04-15T22:24:26.311 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-04-15T22:24:26.311 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-04-15T22:24:26.311 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-04-15T22:24:26.311 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-04-15T22:24:26.311 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-04-15T22:24:26.311 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-04-15T22:24:26.311 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-04-15T22:24:26.311 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-04-15T22:24:26.311 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-04-15T22:24:26.312 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-04-15T22:24:26.312 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-04-15T22:24:26.312 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-04-15T22:24:26.312 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-04-15T22:24:26.312 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-04-15T22:24:26.312 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-04-15T22:24:26.312 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-04-15T22:24:26.312 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-04-15T22:24:26.312 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-04-15T22:24:26.312 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-04-15T22:24:26.312 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-04-15T22:24:26.313 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-04-15T22:24:26.313 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-04-15T22:24:26.313 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-04-15T22:24:26.313 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-04-15T22:24:26.313 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-04-15T22:24:26.313 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-04-15T22:24:26.313 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-04-15T22:24:26.313 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-04-15T22:24:26.314 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-04-15T22:24:26.314 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-15T22:24:26.314 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-15T22:24:26.314 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-15T22:24:26.314 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-04-15T22:24:26.314 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-04-15T22:24:26.314 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-15T22:24:26.314 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-15T22:24:26.315 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-04-15T22:24:26.315 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-15T22:24:26.315 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-04-15T22:24:26.315 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-04-15T22:24:26.316 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-04-15T22:24:26.316 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-15T22:24:26.316 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-04-15T22:24:26.316 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-04-15T22:24:26.316 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-04-15T22:24:26.316 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-04-15T22:24:26.316 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-15T22:24:26.316 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-04-15T22:24:26.316 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-04-15T22:24:26.316 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-04-15T22:24:26.316 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-04-15T22:24:26.317 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-04-15T22:24:26.317 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-15T22:24:26.317 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-04-15T22:24:26.317 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-04-15T22:24:26.317 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-15T22:24:26.317 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-04-15T22:24:26.317 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-15T22:24:26.317 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-15T22:24:26.317 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-15T22:24:26.317 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-04-15T22:24:26.317 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-04-15T22:24:26.317 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-04-15T22:24:26.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-04-15T22:24:26.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-04-15T22:24:26.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-15T22:24:26.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-04-15T22:24:26.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-04-15T22:24:26.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-04-15T22:24:26.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-04-15T22:24:26.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-04-15T22:24:26.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-04-15T22:24:26.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-04-15T22:24:26.318 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-04-15T22:24:26.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-04-15T22:24:26.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-04-15T22:24:26.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-04-15T22:24:26.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-04-15T22:24:26.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-04-15T22:24:26.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-04-15T22:24:26.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-04-15T22:24:26.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-04-15T22:24:26.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-04-15T22:24:26.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-04-15T22:24:26.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-04-15T22:24:26.319 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-04-15T22:24:26.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-04-15T22:24:26.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-04-15T22:24:26.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-04-15T22:24:26.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-04-15T22:24:26.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-04-15T22:24:26.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-04-15T22:24:26.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-04-15T22:24:26.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-04-15T22:24:26.320 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-04-15T22:24:26.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-04-15T22:24:26.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-04-15T22:24:26.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-04-15T22:24:26.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-04-15T22:24:26.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-04-15T22:24:26.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-04-15T22:24:26.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-04-15T22:24:26.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-04-15T22:24:26.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-04-15T22:24:26.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-04-15T22:24:26.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-04-15T22:24:26.321 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-04-15T22:24:26.322 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-04-15T22:24:26.322 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-04-15T22:24:26.322 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=478.237558ms 2024-04-15T22:24:26.322 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-04-15T22:24:26.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-04-15T22:24:26.578 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-04-15T22:24:26.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-04-15T22:24:26.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-04-15T22:24:26.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-04-15T22:24:26.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-04-15T22:24:26.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-15T22:24:26.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-04-15T22:24:26.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-04-15T22:24:26.579 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:24:26+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-04-15T22:24:26.707 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[24813]: osdmap e66: 8 total, 8 up, 8 in 2024-04-15T22:24:26.707 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/126385837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-15T22:24:26.707 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-15T22:24:26.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[24813]: pgmap v59: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-04-15T22:24:26.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:26.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:26.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:26.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:26.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:26.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[24813]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:24:26.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[24813]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-15T22:24:26.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[24813]: Reconfiguring daemon alertmanager.a on smithi092 2024-04-15T22:24:26.708 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:26.708 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[29727]: osdmap e66: 8 total, 8 up, 8 in 2024-04-15T22:24:26.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/126385837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-15T22:24:26.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[29727]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-15T22:24:26.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[29727]: pgmap v59: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-04-15T22:24:26.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:26.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:26.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:26.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:26.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:26.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[29727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:24:26.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[29727]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-15T22:24:26.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[29727]: Reconfiguring daemon alertmanager.a on smithi092 2024-04-15T22:24:26.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:26 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:26.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-mon[25776]: osdmap e66: 8 total, 8 up, 8 in 2024-04-15T22:24:26.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/126385837' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-15T22:24:26.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-mon[25776]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-15T22:24:26.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-mon[25776]: pgmap v59: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-04-15T22:24:26.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:26.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:26.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:26.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:26.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:26.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-mon[25776]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:24:26.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-mon[25776]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-15T22:24:26.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-mon[25776]: Reconfiguring daemon alertmanager.a on smithi092 2024-04-15T22:24:26.979 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:26 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:27.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:27 smithi092 ceph-mon[29727]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-15T22:24:27.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:27 smithi092 ceph-mon[29727]: osdmap e67: 8 total, 8 up, 8 in 2024-04-15T22:24:27.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:27 smithi092 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-15T22:24:27.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:27 smithi092 ceph-mon[24813]: osdmap e67: 8 total, 8 up, 8 in 2024-04-15T22:24:27.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:27 smithi176 ceph-mon[25776]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-15T22:24:27.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:27 smithi176 ceph-mon[25776]: osdmap e67: 8 total, 8 up, 8 in 2024-04-15T22:24:28.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:28 smithi092 ceph-mon[29727]: osdmap e68: 8 total, 8 up, 8 in 2024-04-15T22:24:28.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:28 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-15T22:24:28.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:28 smithi092 ceph-mon[29727]: pgmap v62: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-15T22:24:28.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:28 smithi092 ceph-mon[24813]: osdmap e68: 8 total, 8 up, 8 in 2024-04-15T22:24:28.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:28 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-15T22:24:28.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:28 smithi092 ceph-mon[24813]: pgmap v62: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-15T22:24:28.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:28 smithi176 ceph-mon[25776]: osdmap e68: 8 total, 8 up, 8 in 2024-04-15T22:24:28.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:28 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-15T22:24:28.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:28 smithi176 ceph-mon[25776]: pgmap v62: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-15T22:24:29.464 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:29 smithi092 systemd[1]: Stopping Ceph alertmanager.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:24:29.464 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:29 smithi092 bash[51526]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager.a" found: no such container 2024-04-15T22:24:29.780 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:29 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-15T22:24:29.781 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:29 smithi176 ceph-mon[25776]: osdmap e69: 8 total, 8 up, 8 in 2024-04-15T22:24:29.781 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:29 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-15T22:24:29.802 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:29 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-15T22:24:29.802 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:29 smithi092 ceph-mon[29727]: osdmap e69: 8 total, 8 up, 8 in 2024-04-15T22:24:29.802 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:29 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-15T22:24:29.802 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:29 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[50075]: level=info ts=2024-04-15T22:24:29.496Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-15T22:24:29.803 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:29 smithi092 podman[51539]: 2024-04-15 22:24:29.517310572 +0000 UTC m=+0.137043568 container died b3f677ebd5acad0fe6c37c62d45ece64ed51a501b363f05ccbda400ba3587b92 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:29.803 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:29 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-15T22:24:29.803 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:29 smithi092 ceph-mon[24813]: osdmap e69: 8 total, 8 up, 8 in 2024-04-15T22:24:29.803 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:29 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-15T22:24:30.213 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:29 smithi092 podman[51539]: 2024-04-15 22:24:29.993610655 +0000 UTC m=+0.613343649 container cleanup b3f677ebd5acad0fe6c37c62d45ece64ed51a501b363f05ccbda400ba3587b92 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:30.213 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:29 smithi092 bash[51539]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a 2024-04-15T22:24:30.213 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:30 smithi092 bash[51565]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager.a" found: no such container 2024-04-15T22:24:30.655 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:30 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:24:30] "GET /metrics HTTP/1.1" 200 192605 "" "Prometheus/2.33.4" 2024-04-15T22:24:30.656 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:30 smithi092 podman[51553]: 2024-04-15 22:24:30.403521305 +0000 UTC m=+0.880613990 container remove b3f677ebd5acad0fe6c37c62d45ece64ed51a501b363f05ccbda400ba3587b92 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:30.656 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:30 smithi092 podman[51553]: 2024-04-15 22:24:30.504365224 +0000 UTC m=+0.981457907 volume remove 130b5b58c998f0632cf04f8338487adba00dbe04b759bffd43d6d10d94f56064 2024-04-15T22:24:30.656 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:30 smithi092 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-15T22:24:30.656 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:30 smithi092 systemd[1]: Stopped Ceph alertmanager.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:24:30.656 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:30 smithi092 systemd[1]: Starting Ceph alertmanager.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:24:30.656 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:30 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-15T22:24:30.656 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:30 smithi092 ceph-mon[24813]: osdmap e70: 8 total, 8 up, 8 in 2024-04-15T22:24:30.656 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:30 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-15T22:24:30.656 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:30 smithi092 ceph-mon[24813]: pgmap v65: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-15T22:24:30.657 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:30 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-15T22:24:30.657 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:30 smithi092 ceph-mon[24813]: osdmap e71: 8 total, 8 up, 8 in 2024-04-15T22:24:30.657 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:30 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-15T22:24:30.657 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:30 smithi092 ceph-mon[29727]: osdmap e70: 8 total, 8 up, 8 in 2024-04-15T22:24:30.657 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:30 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-15T22:24:30.657 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:30 smithi092 ceph-mon[29727]: pgmap v65: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-15T22:24:30.657 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:30 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-15T22:24:30.657 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:30 smithi092 ceph-mon[29727]: osdmap e71: 8 total, 8 up, 8 in 2024-04-15T22:24:30.658 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-15T22:24:30.658 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "14132ecf-7ec7-4c4d-a5b2-428d0844667f", 2024-04-15T22:24:30.658 INFO:teuthology.orchestra.run.smithi092.stdout: "epoch": 1, 2024-04-15T22:24:30.658 INFO:teuthology.orchestra.run.smithi092.stdout: "predecessor_uuid": "ee746efb-c2e1-498c-a013-2f5f750a6b5d", 2024-04-15T22:24:30.658 INFO:teuthology.orchestra.run.smithi092.stdout: "sync_status": [], 2024-04-15T22:24:30.658 INFO:teuthology.orchestra.run.smithi092.stdout: "period_map": { 2024-04-15T22:24:30.658 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "14132ecf-7ec7-4c4d-a5b2-428d0844667f", 2024-04-15T22:24:30.658 INFO:teuthology.orchestra.run.smithi092.stdout: "zonegroups": [ 2024-04-15T22:24:30.658 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-15T22:24:30.658 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "861582b5-4be2-42e3-87c8-68fd8cc06060", 2024-04-15T22:24:30.658 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "default", 2024-04-15T22:24:30.658 INFO:teuthology.orchestra.run.smithi092.stdout: "api_name": "default", 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "is_master": "true", 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "endpoints": [], 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "hostnames": [], 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "hostnames_s3website": [], 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "master_zone": "94ef375c-fc99-4597-867f-a8070954ca57", 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "zones": [ 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "id": "94ef375c-fc99-4597-867f-a8070954ca57", 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "z", 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "endpoints": [], 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "log_meta": "false", 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "log_data": "false", 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "bucket_index_max_shards": 11, 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "read_only": "false", 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_type": "", 2024-04-15T22:24:30.659 INFO:teuthology.orchestra.run.smithi092.stdout: "sync_from_all": "true", 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: "sync_from": [], 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: "redirect_zone": "" 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: "placement_targets": [ 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "default-placement", 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: "tags": [], 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: "storage_classes": [ 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: "STANDARD" 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: "default_placement": "default-placement", 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: "realm_id": "335209a9-9b58-4290-9489-e15dd7ac5528", 2024-04-15T22:24:30.660 INFO:teuthology.orchestra.run.smithi092.stdout: "sync_policy": { 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: "groups": [] 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: "short_zone_ids": [ 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: "key": "94ef375c-fc99-4597-867f-a8070954ca57", 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: "val": 4223003394 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: "master_zonegroup": "861582b5-4be2-42e3-87c8-68fd8cc06060", 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: "master_zone": "94ef375c-fc99-4597-867f-a8070954ca57", 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: "period_config": { 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: "bucket_quota": { 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: "enabled": false, 2024-04-15T22:24:30.661 INFO:teuthology.orchestra.run.smithi092.stdout: "check_on_raw": false, 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: "max_size": -1, 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: "max_size_kb": 0, 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: "max_objects": -1 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: "user_quota": { 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: "enabled": false, 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: "check_on_raw": false, 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: "max_size": -1, 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: "max_size_kb": 0, 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: "max_objects": -1 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: "user_ratelimit": { 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: "max_read_ops": 0, 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: "max_write_ops": 0, 2024-04-15T22:24:30.662 INFO:teuthology.orchestra.run.smithi092.stdout: "max_read_bytes": 0, 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: "max_write_bytes": 0, 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: "enabled": false 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: "bucket_ratelimit": { 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: "max_read_ops": 0, 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: "max_write_ops": 0, 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: "max_read_bytes": 0, 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: "max_write_bytes": 0, 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: "enabled": false 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: "anonymous_ratelimit": { 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: "max_read_ops": 0, 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: "max_write_ops": 0, 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: "max_read_bytes": 0, 2024-04-15T22:24:30.663 INFO:teuthology.orchestra.run.smithi092.stdout: "max_write_bytes": 0, 2024-04-15T22:24:30.664 INFO:teuthology.orchestra.run.smithi092.stdout: "enabled": false 2024-04-15T22:24:30.664 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-15T22:24:30.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:24:30.664 INFO:teuthology.orchestra.run.smithi092.stdout: "realm_id": "335209a9-9b58-4290-9489-e15dd7ac5528", 2024-04-15T22:24:30.664 INFO:teuthology.orchestra.run.smithi092.stdout: "realm_name": "r", 2024-04-15T22:24:30.664 INFO:teuthology.orchestra.run.smithi092.stdout: "realm_epoch": 2 2024-04-15T22:24:30.664 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-15T22:24:30.803 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:30 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-15T22:24:30.803 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:30 smithi176 ceph-mon[25776]: osdmap e70: 8 total, 8 up, 8 in 2024-04-15T22:24:30.803 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:30 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-15T22:24:30.803 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:30 smithi176 ceph-mon[25776]: pgmap v65: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.0 KiB/s rd, 511 B/s wr, 7 op/s 2024-04-15T22:24:30.804 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:30 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3570241694' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-15T22:24:30.804 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:30 smithi176 ceph-mon[25776]: osdmap e71: 8 total, 8 up, 8 in 2024-04-15T22:24:31.763 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:31 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:24:31] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:24:31.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-04-15T22:24:32.556 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:32 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:32.556 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:32 smithi092 ceph-mon[24813]: pgmap v67: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:32.557 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:32 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:32.557 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:32 smithi092 ceph-mon[29727]: pgmap v67: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:32.557 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 podman[51720]: 2024-04-15 22:24:32.331421607 +0000 UTC m=+0.020847682 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-15T22:24:32.557 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 podman[51720]: 2024-04-15 22:24:32.440366835 +0000 UTC m=+0.129792910 volume create 8037f3a6b0ee4fb5b0489e078d3c4e25a5aea58aba006d8152a47b804e795a84 2024-04-15T22:24:32.557 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 podman[51720]: 2024-04-15 22:24:32.557502178 +0000 UTC m=+0.246928292 container create 5d76ee1dbbdb2155e8edef2cd4ab4b6731288b897844695e133a26f36c27a507 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:32.712 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:32 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:32.712 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:32 smithi176 ceph-mon[25776]: pgmap v67: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:33.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:33 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:24:33] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:24:33.102 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 podman[51720]: 2024-04-15 22:24:32.85194255 +0000 UTC m=+0.541368626 container init 5d76ee1dbbdb2155e8edef2cd4ab4b6731288b897844695e133a26f36c27a507 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:33.102 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 podman[51720]: 2024-04-15 22:24:32.854923166 +0000 UTC m=+0.544349245 container start 5d76ee1dbbdb2155e8edef2cd4ab4b6731288b897844695e133a26f36c27a507 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:33.103 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=info ts=2024-04-15T22:24:32.864Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-15T22:24:33.103 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=info ts=2024-04-15T22:24:32.864Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-15T22:24:33.103 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=info ts=2024-04-15T22:24:32.865Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.92 port=9094 2024-04-15T22:24:33.103 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=info ts=2024-04-15T22:24:32.866Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-15T22:24:33.103 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=info ts=2024-04-15T22:24:32.892Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-15T22:24:33.103 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=info ts=2024-04-15T22:24:32.892Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-15T22:24:33.103 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=info ts=2024-04-15T22:24:32.894Z caller=main.go:518 msg=Listening address=:9093 2024-04-15T22:24:33.103 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=info ts=2024-04-15T22:24:32.894Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-15T22:24:33.103 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 bash[51720]: 5d76ee1dbbdb2155e8edef2cd4ab4b6731288b897844695e133a26f36c27a507 2024-04-15T22:24:33.103 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:32 smithi092 systemd[1]: Started Ceph alertmanager.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:24:33.979 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:24:33 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:24:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:24:34.033 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:33 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:34.033 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:33 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:34.034 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:33 smithi092 ceph-mon[24813]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-15T22:24:34.034 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:33 smithi092 ceph-mon[24813]: Reconfiguring daemon prometheus.a on smithi176 2024-04-15T22:24:34.034 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:33 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:34.034 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:33 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:34.034 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:33 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:34.034 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:33 smithi092 ceph-mon[29727]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-15T22:24:34.035 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:33 smithi092 ceph-mon[29727]: Reconfiguring daemon prometheus.a on smithi176 2024-04-15T22:24:34.035 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:33 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:34.437 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:33 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:34.437 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:33 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:34.437 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:33 smithi176 ceph-mon[25776]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-15T22:24:34.437 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:33 smithi176 ceph-mon[25776]: Reconfiguring daemon prometheus.a on smithi176 2024-04-15T22:24:34.437 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:33 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:35.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:35 smithi092 ceph-mon[29727]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-15T22:24:35.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:35 smithi092 ceph-mon[24813]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-15T22:24:35.214 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:34 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=info ts=2024-04-15T22:24:34.867Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000146658s 2024-04-15T22:24:35.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:35 smithi176 ceph-mon[25776]: pgmap v68: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-04-15T22:24:36.059 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled rgw.foo update... 2024-04-15T22:24:36.174 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 systemd[1]: Stopping Ceph prometheus.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:24:36.174 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 bash[45549]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus.a" found: no such container 2024-04-15T22:24:36.475 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:24:36.175Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-15T22:24:36.475 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:24:36.175Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-15T22:24:36.475 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:24:36.175Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-15T22:24:36.475 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:24:36.175Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-15T22:24:36.475 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:24:36.175Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-15T22:24:36.475 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:24:36.175Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-15T22:24:36.475 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:24:36.176Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-15T22:24:36.476 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:24:36.176Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-15T22:24:36.476 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:24:36.176Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-15T22:24:36.476 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:24:36.176Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-15T22:24:36.476 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:24:36.176Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-15T22:24:36.476 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[43384]: ts=2024-04-15T22:24:36.176Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-15T22:24:36.476 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 podman[45562]: 2024-04-15 22:24:36.185777981 +0000 UTC m=+0.117954962 container died 69c7526eb7559cbab644bdbcd9c6d0e77583ea052eb89f807f5340d5399756ce (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:36.728 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 podman[45562]: 2024-04-15 22:24:36.67506726 +0000 UTC m=+0.607244271 container cleanup 69c7526eb7559cbab644bdbcd9c6d0e77583ea052eb89f807f5340d5399756ce (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:36.728 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 bash[45562]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a 2024-04-15T22:24:36.728 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 bash[45589]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus.a" found: no such container 2024-04-15T22:24:36.982 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch apply rgw smpl' 2024-04-15T22:24:37.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:37 smithi092 ceph-mon[29727]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 974 B/s wr, 2 op/s 2024-04-15T22:24:37.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:37 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:37.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:37 smithi092 ceph-mon[24813]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 974 B/s wr, 2 op/s 2024-04-15T22:24:37.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:37 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:37.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-mon[25776]: pgmap v69: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 974 B/s rd, 974 B/s wr, 2 op/s 2024-04-15T22:24:37.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:37.228 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 podman[45576]: 2024-04-15 22:24:36.96794199 +0000 UTC m=+0.783655334 container remove 69c7526eb7559cbab644bdbcd9c6d0e77583ea052eb89f807f5340d5399756ce (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:37.229 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-15T22:24:37.229 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 systemd[1]: Stopped Ceph prometheus.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:24:37.229 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:36 smithi176 systemd[1]: Starting Ceph prometheus.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:24:37.229 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 podman[45659]: 2024-04-15 22:24:37.111710496 +0000 UTC m=+0.018426488 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-15T22:24:37.487 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 podman[45659]: 2024-04-15 22:24:37.235299332 +0000 UTC m=+0.142015316 container create 4c2f0c2b64ee94a74ba0409b65531c02eb5cefda674fdb267494acfd9d1000f7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:37.487 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 podman[45659]: 2024-04-15 22:24:37.454949175 +0000 UTC m=+0.361665185 container init 4c2f0c2b64ee94a74ba0409b65531c02eb5cefda674fdb267494acfd9d1000f7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:37.487 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 podman[45659]: 2024-04-15 22:24:37.457992239 +0000 UTC m=+0.364708224 container start 4c2f0c2b64ee94a74ba0409b65531c02eb5cefda674fdb267494acfd9d1000f7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:24:37.487 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.482Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-15T22:24:37.488 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.482Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-15T22:24:37.488 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.482Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-15T22:24:37.488 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.482Z caller=main.go:518 level=info host_details="(Linux 5.14.0-438.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Apr 12 13:55:21 UTC 2024 x86_64 smithi176 (none))" 2024-04-15T22:24:37.488 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.482Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-15T22:24:37.488 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.482Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-15T22:24:37.488 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.484Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-15T22:24:37.488 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.484Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-15T22:24:37.488 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.485Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-15T22:24:37.488 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.488Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-15T22:24:37.488 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.488Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.063µs 2024-04-15T22:24:37.488 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.488Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-15T22:24:37.850 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 bash[45659]: 4c2f0c2b64ee94a74ba0409b65531c02eb5cefda674fdb267494acfd9d1000f7 2024-04-15T22:24:37.850 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 systemd[1]: Started Ceph prometheus.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:24:37.850 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.686Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=2 2024-04-15T22:24:37.851 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.851Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=2 2024-04-15T22:24:37.851 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.851Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=2 maxSegment=2 2024-04-15T22:24:37.851 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.851Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=28.06µs wal_replay_duration=363.458269ms total_replay_duration=363.499442ms 2024-04-15T22:24:38.228 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.854Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-15T22:24:38.228 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.854Z caller=main.go:947 level=info msg="TSDB started" 2024-04-15T22:24:38.228 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.854Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-15T22:24:38.228 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.868Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.443312ms db_storage=770ns remote_storage=1.418µs web_handler=327ns query_engine=879ns scrape=162.53µs scrape_sd=23.474µs notify=18.16µs notify_sd=8.875µs rules=13.999424ms 2024-04-15T22:24:38.228 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:24:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:24:37.868Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-15T22:24:38.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:24:38.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: Saving service rgw.foo spec with placement count:2 2024-04-15T22:24:38.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:24:38.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.92:9093"}]: dispatch 2024-04-15T22:24:38.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:24:38.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:24:38.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.176:3000"}]: dispatch 2024-04-15T22:24:38.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:38.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:38.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:38.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.xmypxq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:24:38.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.xmypxq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:24:38.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.xmypxq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:24:38.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.230 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:38 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:38.259 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:24:38] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: Saving service rgw.foo spec with placement count:2 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.92:9093"}]: dispatch 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.176:3000"}]: dispatch 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.xmypxq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.xmypxq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.xmypxq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='client.14682 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: Saving service rgw.foo spec with placement count:2 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.92:9093"}]: dispatch 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.176:3000"}]: dispatch 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.xmypxq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.xmypxq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi092.xmypxq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:38.262 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:38 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:39.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[29727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:24:39.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[29727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.92:9093"}]: dispatch 2024-04-15T22:24:39.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[29727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:24:39.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[29727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:24:39.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[29727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.176:3000"}]: dispatch 2024-04-15T22:24:39.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[29727]: Saving service rgw.foo spec with placement count:2 2024-04-15T22:24:39.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[29727]: Deploying daemon rgw.foo.smithi092.xmypxq on smithi092 2024-04-15T22:24:39.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[29727]: pgmap v70: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 858 B/s wr, 2 op/s 2024-04-15T22:24:39.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:39.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:39.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:24:39.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.92:9093"}]: dispatch 2024-04-15T22:24:39.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:24:39.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:24:39.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.176:3000"}]: dispatch 2024-04-15T22:24:39.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[24813]: Saving service rgw.foo spec with placement count:2 2024-04-15T22:24:39.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[24813]: Deploying daemon rgw.foo.smithi092.xmypxq on smithi092 2024-04-15T22:24:39.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[24813]: pgmap v70: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 858 B/s wr, 2 op/s 2024-04-15T22:24:39.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:39.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:39 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:39.378 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled rgw.smpl update... 2024-04-15T22:24:39.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:39 smithi176 ceph-mon[25776]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T22:24:39.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:39 smithi176 ceph-mon[25776]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.92:9093"}]: dispatch 2024-04-15T22:24:39.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:39 smithi176 ceph-mon[25776]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T22:24:39.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:39 smithi176 ceph-mon[25776]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T22:24:39.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:39 smithi176 ceph-mon[25776]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.176:3000"}]: dispatch 2024-04-15T22:24:39.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:39 smithi176 ceph-mon[25776]: Saving service rgw.foo spec with placement count:2 2024-04-15T22:24:39.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:39 smithi176 ceph-mon[25776]: Deploying daemon rgw.foo.smithi092.xmypxq on smithi092 2024-04-15T22:24:39.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:39 smithi176 ceph-mon[25776]: pgmap v70: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 858 B/s rd, 858 B/s wr, 2 op/s 2024-04-15T22:24:39.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:39 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:39.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:39 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:40.124 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph osd pool create foo' 2024-04-15T22:24:40.667 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:40 smithi092 ceph-mon[29727]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:24:40.667 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:40 smithi092 ceph-mon[29727]: Saving service rgw.smpl spec with placement count:2 2024-04-15T22:24:40.667 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:40 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:40.668 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:40 smithi092 ceph-mon[29727]: pgmap v71: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-15T22:24:40.668 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:40 smithi092 ceph-mon[24813]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:24:40.668 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:40 smithi092 ceph-mon[24813]: Saving service rgw.smpl spec with placement count:2 2024-04-15T22:24:40.668 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:40 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:40.668 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:40 smithi092 ceph-mon[24813]: pgmap v71: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-15T22:24:40.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:40 smithi176 ceph-mon[25776]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:24:40.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:40 smithi176 ceph-mon[25776]: Saving service rgw.smpl spec with placement count:2 2024-04-15T22:24:40.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:40 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:40.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:40 smithi176 ceph-mon[25776]: pgmap v71: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-15T22:24:42.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:42 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:42.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:42 smithi176 ceph-mon[25776]: pgmap v72: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 631 B/s rd, 631 B/s wr, 1 op/s 2024-04-15T22:24:42.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:42 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:42.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:42 smithi092 ceph-mon[29727]: pgmap v72: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 631 B/s rd, 631 B/s wr, 1 op/s 2024-04-15T22:24:42.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:42 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:42.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:42 smithi092 ceph-mon[24813]: pgmap v72: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 631 B/s rd, 631 B/s wr, 1 op/s 2024-04-15T22:24:42.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:42 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=info ts=2024-04-15T22:24:42.868Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001727678s 2024-04-15T22:24:43.091 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'foo' created 2024-04-15T22:24:43.335 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:43 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3621253617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-15T22:24:43.335 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:43 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3621253617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-15T22:24:43.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:43 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3621253617' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-15T22:24:43.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'rbd pool init foo' 2024-04-15T22:24:44.084 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:24:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:24:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:24:44.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:44 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3621253617' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-15T22:24:44.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:44 smithi092 ceph-mon[29727]: osdmap e72: 8 total, 8 up, 8 in 2024-04-15T22:24:44.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:44 smithi092 ceph-mon[29727]: pgmap v74: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:44.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:44 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3621253617' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-15T22:24:44.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:44 smithi092 ceph-mon[24813]: osdmap e72: 8 total, 8 up, 8 in 2024-04-15T22:24:44.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:44 smithi092 ceph-mon[24813]: pgmap v74: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:44.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:44 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3621253617' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-15T22:24:44.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:44 smithi176 ceph-mon[25776]: osdmap e72: 8 total, 8 up, 8 in 2024-04-15T22:24:44.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:44 smithi176 ceph-mon[25776]: pgmap v74: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:45.345 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:45 smithi092 ceph-mon[24813]: osdmap e73: 8 total, 8 up, 8 in 2024-04-15T22:24:45.345 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:45 smithi092 ceph-mon[29727]: osdmap e73: 8 total, 8 up, 8 in 2024-04-15T22:24:45.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:45 smithi176 ceph-mon[25776]: osdmap e73: 8 total, 8 up, 8 in 2024-04-15T22:24:46.286 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:46 smithi092 ceph-mon[29727]: osdmap e74: 8 total, 8 up, 8 in 2024-04-15T22:24:46.286 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:46 smithi092 ceph-mon[29727]: pgmap v77: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:46.286 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:24:46.287 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:24:46.287 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:24:46.287 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:24:46.287 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:46 smithi092 ceph-mon[24813]: osdmap e74: 8 total, 8 up, 8 in 2024-04-15T22:24:46.287 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:46 smithi092 ceph-mon[24813]: pgmap v77: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:46.287 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:24:46.287 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:24:46.287 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:24:46.287 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:24:46.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:46 smithi176 ceph-mon[25776]: osdmap e74: 8 total, 8 up, 8 in 2024-04-15T22:24:46.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:46 smithi176 ceph-mon[25776]: pgmap v77: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:24:46.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:24:46.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:24:46.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:24:46.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:24:47.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2606928631' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-15T22:24:47.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[29727]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-15T22:24:47.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:47.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi176.twznrb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:24:47.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi176.twznrb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:24:47.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi176.twznrb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:24:47.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:47.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:47.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[29727]: Deploying daemon rgw.foo.smithi176.twznrb on smithi176 2024-04-15T22:24:47.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:47.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2606928631' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-15T22:24:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-15T22:24:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi176.twznrb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:24:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi176.twznrb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:24:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi176.twznrb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:24:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[24813]: Deploying daemon rgw.foo.smithi176.twznrb on smithi176 2024-04-15T22:24:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:47.215 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:47.215 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:46 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:24:46] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:24:47.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:47 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2606928631' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-15T22:24:47.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:47 smithi176 ceph-mon[25776]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-15T22:24:47.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:47.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:47 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi176.twznrb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:24:47.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi176.twznrb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:24:47.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi176.twznrb", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:24:47.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:47.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:47 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:47.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:47 smithi176 ceph-mon[25776]: Deploying daemon rgw.foo.smithi176.twznrb on smithi176 2024-04-15T22:24:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:47.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:48.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:48 smithi092 ceph-mon[29727]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-15T22:24:48.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:48 smithi092 ceph-mon[29727]: osdmap e75: 8 total, 8 up, 8 in 2024-04-15T22:24:48.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:48 smithi092 ceph-mon[29727]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 6.8 KiB/s wr, 230 op/s 2024-04-15T22:24:48.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:48 smithi092 ceph-mon[24813]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-15T22:24:48.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:48 smithi092 ceph-mon[24813]: osdmap e75: 8 total, 8 up, 8 in 2024-04-15T22:24:48.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:48 smithi092 ceph-mon[24813]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 6.8 KiB/s wr, 230 op/s 2024-04-15T22:24:48.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:48 smithi176 ceph-mon[25776]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-15T22:24:48.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:48 smithi176 ceph-mon[25776]: osdmap e75: 8 total, 8 up, 8 in 2024-04-15T22:24:48.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:48 smithi176 ceph-mon[25776]: pgmap v79: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 95 KiB/s rd, 6.8 KiB/s wr, 230 op/s 2024-04-15T22:24:49.422 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:49 smithi176 ceph-mon[25776]: osdmap e76: 8 total, 8 up, 8 in 2024-04-15T22:24:49.450 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:49 smithi092 ceph-mon[29727]: osdmap e76: 8 total, 8 up, 8 in 2024-04-15T22:24:49.451 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:49 smithi092 ceph-mon[24813]: osdmap e76: 8 total, 8 up, 8 in 2024-04-15T22:24:49.829 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch apply iscsi foo u p' 2024-04-15T22:24:50.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:50 smithi092 ceph-mon[24813]: osdmap e77: 8 total, 8 up, 8 in 2024-04-15T22:24:50.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:50 smithi092 ceph-mon[24813]: pgmap v82: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 6.8 KiB/s wr, 230 op/s 2024-04-15T22:24:50.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:50 smithi092 ceph-mon[29727]: osdmap e77: 8 total, 8 up, 8 in 2024-04-15T22:24:50.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:50 smithi092 ceph-mon[29727]: pgmap v82: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 6.8 KiB/s wr, 230 op/s 2024-04-15T22:24:50.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:50 smithi176 ceph-mon[25776]: osdmap e77: 8 total, 8 up, 8 in 2024-04-15T22:24:50.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:50 smithi176 ceph-mon[25776]: pgmap v82: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s rd, 6.8 KiB/s wr, 230 op/s 2024-04-15T22:24:50.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:24:50] "GET /metrics HTTP/1.1" 200 197865 "" "Prometheus/2.33.4" 2024-04-15T22:24:53.405 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:53 smithi176 ceph-mon[25776]: pgmap v83: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.5 KiB/s wr, 186 op/s 2024-04-15T22:24:53.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:53 smithi092 ceph-mon[29727]: pgmap v83: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.5 KiB/s wr, 186 op/s 2024-04-15T22:24:53.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:53 smithi092 ceph-mon[24813]: pgmap v83: 161 pgs: 161 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.5 KiB/s wr, 186 op/s 2024-04-15T22:24:53.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:24:53] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:24:53.833 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:24:53.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:24:53.833 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:24:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:24:53.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:24:53.978 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:24:53 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:24:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:24:54.173 INFO:teuthology.orchestra.run.smithi092.stdout:Scheduled iscsi.foo update... 2024-04-15T22:24:54.433 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.433 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.433 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.433 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:54.433 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:54.433 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:54.433 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: pgmap v84: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 321 B/s wr, 0 op/s 2024-04-15T22:24:54.433 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: Checking dashboard <-> RGW credentials 2024-04-15T22:24:54.433 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.433 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.434 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.434 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:24:54.434 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:54.434 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:54.434 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:54.434 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: pgmap v85: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 348 B/s wr, 0 op/s 2024-04-15T22:24:54.434 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: pgmap v86: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 358 B/s wr, 0 op/s 2024-04-15T22:24:54.434 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: Checking dashboard <-> RGW credentials 2024-04-15T22:24:54.434 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.434 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[24813]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:24:54.434 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.434 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: pgmap v84: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 321 B/s wr, 0 op/s 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: Checking dashboard <-> RGW credentials 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: pgmap v85: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 348 B/s wr, 0 op/s 2024-04-15T22:24:54.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: pgmap v86: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 358 B/s wr, 0 op/s 2024-04-15T22:24:54.436 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: Checking dashboard <-> RGW credentials 2024-04-15T22:24:54.436 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.436 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:54 smithi092 ceph-mon[29727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:24:54.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:54.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: pgmap v84: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 321 B/s wr, 0 op/s 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: Checking dashboard <-> RGW credentials 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: pgmap v85: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 348 B/s wr, 0 op/s 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: pgmap v86: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 358 B/s wr, 0 op/s 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: Checking dashboard <-> RGW credentials 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:54.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:54 smithi176 ceph-mon[25776]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:24:54.898 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'sleep 120' 2024-04-15T22:24:55.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[29727]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:24:55.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[29727]: Saving service iscsi.foo spec with placement count:1 2024-04-15T22:24:55.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:55.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:55.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:55.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:55.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[29727]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 402 B/s wr, 0 op/s 2024-04-15T22:24:55.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[29727]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 481 B/s wr, 0 op/s 2024-04-15T22:24:55.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:55.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[29727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:24:55.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[29727]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:24:55.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[24813]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:24:55.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[24813]: Saving service iscsi.foo spec with placement count:1 2024-04-15T22:24:55.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:55.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:55.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:55.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:55.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[24813]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 402 B/s wr, 0 op/s 2024-04-15T22:24:55.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[24813]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 481 B/s wr, 0 op/s 2024-04-15T22:24:55.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:55.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[24813]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:24:55.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:55 smithi092 ceph-mon[24813]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:24:55.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:55 smithi176 ceph-mon[25776]: from='client.14805 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:24:55.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:55 smithi176 ceph-mon[25776]: Saving service iscsi.foo spec with placement count:1 2024-04-15T22:24:55.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:55 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:55.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:55 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:24:55.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:55 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:24:55.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:55 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:24:55.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:55 smithi176 ceph-mon[25776]: pgmap v87: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 402 B/s wr, 0 op/s 2024-04-15T22:24:55.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:55 smithi176 ceph-mon[25776]: pgmap v88: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 481 B/s wr, 0 op/s 2024-04-15T22:24:55.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:55 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:55.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:55 smithi176 ceph-mon[25776]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:24:55.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:55 smithi176 ceph-mon[25776]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-15T22:24:57.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:57 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:57.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:57 smithi176 ceph-mon[25776]: pgmap v89: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 481 B/s wr, 0 op/s 2024-04-15T22:24:57.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:57 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:57.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:57 smithi092 ceph-mon[29727]: pgmap v89: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 481 B/s wr, 0 op/s 2024-04-15T22:24:57.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:57 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:57.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:57 smithi092 ceph-mon[24813]: pgmap v89: 161 pgs: 161 active+clean; 584 KiB data, 55 MiB used, 715 GiB / 715 GiB avail; 481 B/s wr, 0 op/s 2024-04-15T22:24:58.894 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:58 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:24:58] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:24:59.781 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:59 smithi092 ceph-mon[24813]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 153 KiB/s rd, 651 B/s wr, 228 op/s 2024-04-15T22:24:59.781 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:59 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:59.781 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:24:59 smithi092 ceph-mon[24813]: osdmap e78: 8 total, 8 up, 8 in 2024-04-15T22:24:59.781 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:59 smithi092 ceph-mon[29727]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 153 KiB/s rd, 651 B/s wr, 228 op/s 2024-04-15T22:24:59.781 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:59 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:59.781 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:24:59 smithi092 ceph-mon[29727]: osdmap e78: 8 total, 8 up, 8 in 2024-04-15T22:24:59.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:59 smithi176 ceph-mon[25776]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 153 KiB/s rd, 651 B/s wr, 228 op/s 2024-04-15T22:24:59.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:59 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:24:59.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:24:59 smithi176 ceph-mon[25776]: osdmap e78: 8 total, 8 up, 8 in 2024-04-15T22:25:00.214 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:24:59 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:24:59] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:00.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:00 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:00.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:00 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:00.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:25:00] "GET /metrics HTTP/1.1" 200 197865 "" "Prometheus/2.33.4" 2024-04-15T22:25:01.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:00 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:01.606 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:25:01] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:01.606 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:25:01] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:01.606 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:25:01] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:02.331 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 481 B/s wr, 169 op/s 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.juhhhh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.juhhhh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:02.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.juhhhh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:25:02.333 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.333 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:02.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 481 B/s wr, 169 op/s 2024-04-15T22:25:02.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:02.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:02.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:02.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.334 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.334 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.juhhhh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:02.334 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.juhhhh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:02.334 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.juhhhh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:25:02.334 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.334 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:01 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:02.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: pgmap v92: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 113 KiB/s rd, 481 B/s wr, 169 op/s 2024-04-15T22:25:02.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.juhhhh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.juhhhh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi092.juhhhh", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:25:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:02.479 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:01 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:03.297 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:02 smithi092 ceph-mon[29727]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 418 B/s wr, 146 op/s 2024-04-15T22:25:03.297 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:02 smithi092 ceph-mon[29727]: Saving service rgw.smpl spec with placement count:2 2024-04-15T22:25:03.297 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:02 smithi092 ceph-mon[29727]: Deploying daemon rgw.smpl.smithi092.juhhhh on smithi092 2024-04-15T22:25:03.298 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:02 smithi092 ceph-mon[29727]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:25:03.298 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:02 smithi092 ceph-mon[29727]: Cluster is now healthy 2024-04-15T22:25:03.298 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:02 smithi092 ceph-mon[24813]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 418 B/s wr, 146 op/s 2024-04-15T22:25:03.298 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:02 smithi092 ceph-mon[24813]: Saving service rgw.smpl spec with placement count:2 2024-04-15T22:25:03.298 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:02 smithi092 ceph-mon[24813]: Deploying daemon rgw.smpl.smithi092.juhhhh on smithi092 2024-04-15T22:25:03.298 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:02 smithi092 ceph-mon[24813]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:25:03.298 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:02 smithi092 ceph-mon[24813]: Cluster is now healthy 2024-04-15T22:25:03.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:02 smithi176 ceph-mon[25776]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s rd, 418 B/s wr, 146 op/s 2024-04-15T22:25:03.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:02 smithi176 ceph-mon[25776]: Saving service rgw.smpl spec with placement count:2 2024-04-15T22:25:03.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:02 smithi176 ceph-mon[25776]: Deploying daemon rgw.smpl.smithi092.juhhhh on smithi092 2024-04-15T22:25:03.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:02 smithi176 ceph-mon[25776]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-15T22:25:03.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:02 smithi176 ceph-mon[25776]: Cluster is now healthy 2024-04-15T22:25:03.630 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:25:03.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:25:03.630 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:25:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:25:03.630 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:25:03.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:25:04.127 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:25:03 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:25:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:25:05.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:04 smithi176 ceph-mon[25776]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 328 B/s wr, 115 op/s 2024-04-15T22:25:05.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:04 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:05.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:04 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi176.fwngry", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:05.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:04 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi176.fwngry", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:05.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:04 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi176.fwngry", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:25:05.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:04 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:05.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:04 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:05.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[24813]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 328 B/s wr, 115 op/s 2024-04-15T22:25:05.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:05.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi176.fwngry", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:05.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi176.fwngry", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:05.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi176.fwngry", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:25:05.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:05.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:05.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[29727]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 328 B/s wr, 115 op/s 2024-04-15T22:25:05.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:05.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi176.fwngry", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:05.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi176.fwngry", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T22:25:05.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi176.fwngry", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T22:25:05.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:05.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:04 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:06.149 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:06 smithi176 ceph-mon[25776]: Deploying daemon rgw.smpl.smithi176.fwngry on smithi176 2024-04-15T22:25:06.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:06 smithi092 ceph-mon[24813]: Deploying daemon rgw.smpl.smithi176.fwngry on smithi176 2024-04-15T22:25:06.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:06 smithi092 ceph-mon[29727]: Deploying daemon rgw.smpl.smithi176.fwngry on smithi176 2024-04-15T22:25:07.171 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:07 smithi176 ceph-mon[25776]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 328 B/s wr, 115 op/s 2024-04-15T22:25:07.171 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:07 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:07.171 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:07 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.hsbujv", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-15T22:25:07.171 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:07 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.hsbujv", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-15T22:25:07.171 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:07 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.hsbujv", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-15T22:25:07.171 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:07 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:07.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:07 smithi092 ceph-mon[29727]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 328 B/s wr, 115 op/s 2024-04-15T22:25:07.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:07 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:07.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:07 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.hsbujv", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-15T22:25:07.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:07 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.hsbujv", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-15T22:25:07.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:07 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.hsbujv", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-15T22:25:07.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:07 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:07.262 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:07 smithi092 ceph-mon[24813]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 61 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 328 B/s wr, 115 op/s 2024-04-15T22:25:07.262 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:07 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:07.262 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:07 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.hsbujv", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-15T22:25:07.262 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:07 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.hsbujv", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-15T22:25:07.262 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:07 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi092.hsbujv", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-15T22:25:07.262 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:07 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:08.168 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:08 smithi176 ceph-mon[25776]: Checking pool "foo" exists for service iscsi.foo 2024-04-15T22:25:08.169 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:08 smithi176 ceph-mon[25776]: Deploying daemon iscsi.foo.smithi092.hsbujv on smithi092 2024-04-15T22:25:08.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:08 smithi092 ceph-mon[29727]: Checking pool "foo" exists for service iscsi.foo 2024-04-15T22:25:08.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:08 smithi092 ceph-mon[29727]: Deploying daemon iscsi.foo.smithi092.hsbujv on smithi092 2024-04-15T22:25:08.371 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:08 smithi092 ceph-mon[24813]: Checking pool "foo" exists for service iscsi.foo 2024-04-15T22:25:08.371 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:08 smithi092 ceph-mon[24813]: Deploying daemon iscsi.foo.smithi092.hsbujv on smithi092 2024-04-15T22:25:09.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:09 smithi176 ceph-mon[25776]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 219 B/s wr, 215 op/s 2024-04-15T22:25:09.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:09 smithi176 ceph-mon[25776]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-15T22:25:09.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:09 smithi092 ceph-mon[29727]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 219 B/s wr, 215 op/s 2024-04-15T22:25:09.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:09 smithi092 ceph-mon[29727]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-15T22:25:09.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:09 smithi092 ceph-mon[24813]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 139 KiB/s rd, 219 B/s wr, 215 op/s 2024-04-15T22:25:09.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:09 smithi092 ceph-mon[24813]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-15T22:25:10.072 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:10 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:25:10] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:10.607 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:10 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:25:10] "GET /metrics HTTP/1.1" 200 208150 "" "Prometheus/2.33.4" 2024-04-15T22:25:11.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 197 B/s wr, 193 op/s 2024-04-15T22:25:11.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:11.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:11.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:11.334 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-15T22:25:11.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-15T22:25:11.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi092"}]: dispatch 2024-04-15T22:25:11.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:11.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:11.335 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:11 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:11.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 197 B/s wr, 193 op/s 2024-04-15T22:25:11.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:11.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:11.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:11.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-15T22:25:11.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-15T22:25:11.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi092"}]: dispatch 2024-04-15T22:25:11.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.439 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:11.439 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:11.439 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:11.439 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s rd, 197 B/s wr, 193 op/s 2024-04-15T22:25:11.439 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.439 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.439 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:11.439 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:11.439 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:11.439 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.439 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-15T22:25:11.440 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-15T22:25:11.440 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.440 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi092"}]: dispatch 2024-04-15T22:25:11.440 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:11.440 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:11.440 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:11.440 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:11 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:12.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:12 smithi176 ceph-mon[25776]: Checking dashboard <-> RGW credentials 2024-04-15T22:25:12.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:12 smithi176 ceph-mon[25776]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:25:12.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:12 smithi176 ceph-mon[25776]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-15T22:25:12.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:12 smithi176 ceph-mon[25776]: Adding iSCSI gateway http://:@172.21.15.92:5000 to Dashboard 2024-04-15T22:25:12.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:12 smithi176 ceph-mon[25776]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-15T22:25:12.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:12 smithi176 ceph-mon[25776]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi092"}]: dispatch 2024-04-15T22:25:12.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:12 smithi176 ceph-mon[25776]: Checking dashboard <-> RGW credentials 2024-04-15T22:25:12.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:12 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:12.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:12 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:12.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:12 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3284258030' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-15T22:25:12.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:12 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2995073700' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4139644651"}]: dispatch 2024-04-15T22:25:12.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:12 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4139644651"}]: dispatch 2024-04-15T22:25:12.309 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[29727]: Checking dashboard <-> RGW credentials 2024-04-15T22:25:12.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[29727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:25:12.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[29727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-15T22:25:12.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[29727]: Adding iSCSI gateway http://:@172.21.15.92:5000 to Dashboard 2024-04-15T22:25:12.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[29727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-15T22:25:12.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[29727]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi092"}]: dispatch 2024-04-15T22:25:12.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[29727]: Checking dashboard <-> RGW credentials 2024-04-15T22:25:12.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:12.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:12.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3284258030' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-15T22:25:12.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2995073700' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4139644651"}]: dispatch 2024-04-15T22:25:12.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4139644651"}]: dispatch 2024-04-15T22:25:12.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[24813]: Checking dashboard <-> RGW credentials 2024-04-15T22:25:12.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[24813]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:25:12.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-15T22:25:12.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[24813]: Adding iSCSI gateway http://:@172.21.15.92:5000 to Dashboard 2024-04-15T22:25:12.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-15T22:25:12.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[24813]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi092"}]: dispatch 2024-04-15T22:25:12.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[24813]: Checking dashboard <-> RGW credentials 2024-04-15T22:25:12.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:12.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:12.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3284258030' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-15T22:25:12.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2995073700' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4139644651"}]: dispatch 2024-04-15T22:25:12.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4139644651"}]: dispatch 2024-04-15T22:25:12.713 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:25:12] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:13.009 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:12 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:25:12] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:13.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[24813]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:25:13.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[24813]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 180 B/s wr, 177 op/s 2024-04-15T22:25:13.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4139644651"}]': finished 2024-04-15T22:25:13.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[24813]: osdmap e79: 8 total, 8 up, 8 in 2024-04-15T22:25:13.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:13.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:13.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3763860605' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1886766826"}]: dispatch 2024-04-15T22:25:13.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:13.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[29727]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:25:13.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[29727]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 180 B/s wr, 177 op/s 2024-04-15T22:25:13.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4139644651"}]': finished 2024-04-15T22:25:13.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[29727]: osdmap e79: 8 total, 8 up, 8 in 2024-04-15T22:25:13.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:13.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:13.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3763860605' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1886766826"}]: dispatch 2024-04-15T22:25:13.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:13.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:13 smithi176 ceph-mon[25776]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-15T22:25:13.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:13 smithi176 ceph-mon[25776]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 87 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s rd, 180 B/s wr, 177 op/s 2024-04-15T22:25:13.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:13 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4139644651"}]': finished 2024-04-15T22:25:13.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:13 smithi176 ceph-mon[25776]: osdmap e79: 8 total, 8 up, 8 in 2024-04-15T22:25:13.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:13 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:13.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:13 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:13.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:13 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3763860605' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1886766826"}]: dispatch 2024-04-15T22:25:13.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:13 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:13.713 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:25:13.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:25:13.714 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:25:13.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:25:13.714 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:25:13.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:25:14.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:25:13 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:25:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:25:14.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:14 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3763860605' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1886766826"}]': finished 2024-04-15T22:25:14.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:14 smithi092 ceph-mon[29727]: osdmap e80: 8 total, 8 up, 8 in 2024-04-15T22:25:14.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:14 smithi092 ceph-mon[29727]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 383 B/s wr, 261 op/s 2024-04-15T22:25:14.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:14 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3097178527' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/396238662"}]: dispatch 2024-04-15T22:25:14.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:14 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3763860605' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1886766826"}]': finished 2024-04-15T22:25:14.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:14 smithi092 ceph-mon[24813]: osdmap e80: 8 total, 8 up, 8 in 2024-04-15T22:25:14.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:14 smithi092 ceph-mon[24813]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 383 B/s wr, 261 op/s 2024-04-15T22:25:14.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:14 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3097178527' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/396238662"}]: dispatch 2024-04-15T22:25:14.731 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:14 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3763860605' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1886766826"}]': finished 2024-04-15T22:25:14.731 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:14 smithi176 ceph-mon[25776]: osdmap e80: 8 total, 8 up, 8 in 2024-04-15T22:25:14.731 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:14 smithi176 ceph-mon[25776]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 171 KiB/s rd, 383 B/s wr, 261 op/s 2024-04-15T22:25:14.731 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:14 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3097178527' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/396238662"}]: dispatch 2024-04-15T22:25:15.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:15 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3097178527' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/396238662"}]': finished 2024-04-15T22:25:15.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:15 smithi092 ceph-mon[29727]: osdmap e81: 8 total, 8 up, 8 in 2024-04-15T22:25:15.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:15 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2236049793' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/379636010"}]: dispatch 2024-04-15T22:25:15.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:15 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3097178527' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/396238662"}]': finished 2024-04-15T22:25:15.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:15 smithi092 ceph-mon[24813]: osdmap e81: 8 total, 8 up, 8 in 2024-04-15T22:25:15.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:15 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2236049793' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/379636010"}]: dispatch 2024-04-15T22:25:15.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:15 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3097178527' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/396238662"}]': finished 2024-04-15T22:25:15.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:15 smithi176 ceph-mon[25776]: osdmap e81: 8 total, 8 up, 8 in 2024-04-15T22:25:15.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:15 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2236049793' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/379636010"}]: dispatch 2024-04-15T22:25:16.378 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:25:16] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:16.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2236049793' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/379636010"}]': finished 2024-04-15T22:25:16.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[24813]: osdmap e82: 8 total, 8 up, 8 in 2024-04-15T22:25:16.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[24813]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 255 B/s wr, 19 op/s 2024-04-15T22:25:16.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3676833419' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4084236890"}]: dispatch 2024-04-15T22:25:16.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4084236890"}]: dispatch 2024-04-15T22:25:16.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:16.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:16.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2236049793' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/379636010"}]': finished 2024-04-15T22:25:16.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[29727]: osdmap e82: 8 total, 8 up, 8 in 2024-04-15T22:25:16.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[29727]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 255 B/s wr, 19 op/s 2024-04-15T22:25:16.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3676833419' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4084236890"}]: dispatch 2024-04-15T22:25:16.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4084236890"}]: dispatch 2024-04-15T22:25:16.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:16.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:16 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:16.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:16 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2236049793' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/379636010"}]': finished 2024-04-15T22:25:16.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:16 smithi176 ceph-mon[25776]: osdmap e82: 8 total, 8 up, 8 in 2024-04-15T22:25:16.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:16 smithi176 ceph-mon[25776]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 255 B/s wr, 19 op/s 2024-04-15T22:25:16.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:16 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3676833419' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4084236890"}]: dispatch 2024-04-15T22:25:16.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:16 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4084236890"}]: dispatch 2024-04-15T22:25:16.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:16 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:16.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:16 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:17.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:17 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4084236890"}]': finished 2024-04-15T22:25:17.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:17 smithi092 ceph-mon[29727]: osdmap e83: 8 total, 8 up, 8 in 2024-04-15T22:25:17.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:17 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/310954351' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4084236890"}]: dispatch 2024-04-15T22:25:17.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:17 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4084236890"}]: dispatch 2024-04-15T22:25:17.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:17 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4084236890"}]': finished 2024-04-15T22:25:17.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:17 smithi092 ceph-mon[24813]: osdmap e83: 8 total, 8 up, 8 in 2024-04-15T22:25:17.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:17 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/310954351' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4084236890"}]: dispatch 2024-04-15T22:25:17.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:17 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4084236890"}]: dispatch 2024-04-15T22:25:17.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:17 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/4084236890"}]': finished 2024-04-15T22:25:17.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:17 smithi176 ceph-mon[25776]: osdmap e83: 8 total, 8 up, 8 in 2024-04-15T22:25:17.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:17 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/310954351' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4084236890"}]: dispatch 2024-04-15T22:25:17.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:17 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4084236890"}]: dispatch 2024-04-15T22:25:18.026 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:17 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:25:17] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:18.026 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:25:18] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:18.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4084236890"}]': finished 2024-04-15T22:25:18.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[29727]: osdmap e84: 8 total, 8 up, 8 in 2024-04-15T22:25:18.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[29727]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:18.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3378855951' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2343244098"}]: dispatch 2024-04-15T22:25:18.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2343244098"}]: dispatch 2024-04-15T22:25:18.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:18.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:18.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:18.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:18.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:18.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:18.715 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:18.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4084236890"}]': finished 2024-04-15T22:25:18.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[24813]: osdmap e84: 8 total, 8 up, 8 in 2024-04-15T22:25:18.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[24813]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:18.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3378855951' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2343244098"}]: dispatch 2024-04-15T22:25:18.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2343244098"}]: dispatch 2024-04-15T22:25:18.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:18.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:18.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:18.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:18.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:18.716 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:18.716 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:18 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:18.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:18 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4084236890"}]': finished 2024-04-15T22:25:18.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:18 smithi176 ceph-mon[25776]: osdmap e84: 8 total, 8 up, 8 in 2024-04-15T22:25:18.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:18 smithi176 ceph-mon[25776]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:18.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:18 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3378855951' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2343244098"}]: dispatch 2024-04-15T22:25:18.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:18 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2343244098"}]: dispatch 2024-04-15T22:25:18.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:18 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:18.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:18 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:18.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:18 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:18.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:18 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:25:18.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:18 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:25:18.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:18 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:25:18.729 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:18 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:19.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:19 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2343244098"}]': finished 2024-04-15T22:25:19.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:19 smithi092 ceph-mon[24813]: osdmap e85: 8 total, 8 up, 8 in 2024-04-15T22:25:19.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:19 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/1809057419' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4139644651"}]: dispatch 2024-04-15T22:25:19.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:19 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2343244098"}]': finished 2024-04-15T22:25:19.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:19 smithi092 ceph-mon[29727]: osdmap e85: 8 total, 8 up, 8 in 2024-04-15T22:25:19.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:19 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/1809057419' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4139644651"}]: dispatch 2024-04-15T22:25:19.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:19 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/2343244098"}]': finished 2024-04-15T22:25:19.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:19 smithi176 ceph-mon[25776]: osdmap e85: 8 total, 8 up, 8 in 2024-04-15T22:25:19.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:19 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/1809057419' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4139644651"}]: dispatch 2024-04-15T22:25:20.713 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:20 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:25:20] "GET /metrics HTTP/1.1" 200 215035 "" "Prometheus/2.33.4" 2024-04-15T22:25:20.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:20 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/1809057419' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4139644651"}]': finished 2024-04-15T22:25:20.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:20 smithi092 ceph-mon[29727]: osdmap e86: 8 total, 8 up, 8 in 2024-04-15T22:25:20.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:20 smithi092 ceph-mon[29727]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:20.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:20 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2663849856' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1349181575"}]: dispatch 2024-04-15T22:25:20.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:20 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1349181575"}]: dispatch 2024-04-15T22:25:20.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:20 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/1809057419' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4139644651"}]': finished 2024-04-15T22:25:20.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:20 smithi092 ceph-mon[24813]: osdmap e86: 8 total, 8 up, 8 in 2024-04-15T22:25:20.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:20 smithi092 ceph-mon[24813]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:20.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:20 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2663849856' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1349181575"}]: dispatch 2024-04-15T22:25:20.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:20 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1349181575"}]: dispatch 2024-04-15T22:25:20.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:20 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/1809057419' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/4139644651"}]': finished 2024-04-15T22:25:20.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:20 smithi176 ceph-mon[25776]: osdmap e86: 8 total, 8 up, 8 in 2024-04-15T22:25:20.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:20 smithi176 ceph-mon[25776]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:20.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:20 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2663849856' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1349181575"}]: dispatch 2024-04-15T22:25:20.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:20 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1349181575"}]: dispatch 2024-04-15T22:25:21.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:21 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1349181575"}]': finished 2024-04-15T22:25:21.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:21 smithi092 ceph-mon[24813]: osdmap e87: 8 total, 8 up, 8 in 2024-04-15T22:25:21.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:21 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2196248654' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/3203283767"}]: dispatch 2024-04-15T22:25:21.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:21 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/3203283767"}]: dispatch 2024-04-15T22:25:21.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:21 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:25:21.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:21 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1349181575"}]': finished 2024-04-15T22:25:21.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:21 smithi092 ceph-mon[29727]: osdmap e87: 8 total, 8 up, 8 in 2024-04-15T22:25:21.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:21 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2196248654' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/3203283767"}]: dispatch 2024-04-15T22:25:21.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:21 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/3203283767"}]: dispatch 2024-04-15T22:25:21.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:21 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:25:21.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:21 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1349181575"}]': finished 2024-04-15T22:25:21.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:21 smithi176 ceph-mon[25776]: osdmap e87: 8 total, 8 up, 8 in 2024-04-15T22:25:21.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:21 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2196248654' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/3203283767"}]: dispatch 2024-04-15T22:25:21.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:21 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/3203283767"}]: dispatch 2024-04-15T22:25:21.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:21 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:25:22.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:22 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/3203283767"}]': finished 2024-04-15T22:25:22.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:22 smithi092 ceph-mon[29727]: osdmap e88: 8 total, 8 up, 8 in 2024-04-15T22:25:22.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:22 smithi092 ceph-mon[29727]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:22.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:22 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/1310637552' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3300011615"}]: dispatch 2024-04-15T22:25:22.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:22 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3300011615"}]: dispatch 2024-04-15T22:25:22.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:22 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/3203283767"}]': finished 2024-04-15T22:25:22.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:22 smithi092 ceph-mon[24813]: osdmap e88: 8 total, 8 up, 8 in 2024-04-15T22:25:22.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:22 smithi092 ceph-mon[24813]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:22.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:22 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/1310637552' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3300011615"}]: dispatch 2024-04-15T22:25:22.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:22 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3300011615"}]: dispatch 2024-04-15T22:25:22.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:22 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6800/3203283767"}]': finished 2024-04-15T22:25:22.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:22 smithi176 ceph-mon[25776]: osdmap e88: 8 total, 8 up, 8 in 2024-04-15T22:25:22.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:22 smithi176 ceph-mon[25776]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:22.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:22 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/1310637552' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3300011615"}]: dispatch 2024-04-15T22:25:22.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:22 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3300011615"}]: dispatch 2024-04-15T22:25:23.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:23 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3300011615"}]': finished 2024-04-15T22:25:23.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:23 smithi092 ceph-mon[29727]: osdmap e89: 8 total, 8 up, 8 in 2024-04-15T22:25:23.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:23 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/28953117' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1609306583"}]: dispatch 2024-04-15T22:25:23.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:23 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1609306583"}]: dispatch 2024-04-15T22:25:23.714 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:23 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:25:23.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:25:23.714 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:23 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:25:23.510Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:25:23.714 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:23 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:25:23.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:25:23.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:23 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3300011615"}]': finished 2024-04-15T22:25:23.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:23 smithi092 ceph-mon[24813]: osdmap e89: 8 total, 8 up, 8 in 2024-04-15T22:25:23.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:23 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/28953117' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1609306583"}]: dispatch 2024-04-15T22:25:23.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:23 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1609306583"}]: dispatch 2024-04-15T22:25:23.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:23 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3300011615"}]': finished 2024-04-15T22:25:23.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:23 smithi176 ceph-mon[25776]: osdmap e89: 8 total, 8 up, 8 in 2024-04-15T22:25:23.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:23 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/28953117' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1609306583"}]: dispatch 2024-04-15T22:25:23.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:23 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1609306583"}]: dispatch 2024-04-15T22:25:24.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:25:23 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:25:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:25:24.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:24 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1609306583"}]': finished 2024-04-15T22:25:24.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:24 smithi092 ceph-mon[24813]: osdmap e90: 8 total, 8 up, 8 in 2024-04-15T22:25:24.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:24 smithi092 ceph-mon[24813]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:24.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:24 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/313648142' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/3203283767"}]: dispatch 2024-04-15T22:25:24.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:24 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1609306583"}]': finished 2024-04-15T22:25:24.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:24 smithi092 ceph-mon[29727]: osdmap e90: 8 total, 8 up, 8 in 2024-04-15T22:25:24.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:24 smithi092 ceph-mon[29727]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:24.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:24 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/313648142' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/3203283767"}]: dispatch 2024-04-15T22:25:24.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:24 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1609306583"}]': finished 2024-04-15T22:25:24.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:24 smithi176 ceph-mon[25776]: osdmap e90: 8 total, 8 up, 8 in 2024-04-15T22:25:24.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:24 smithi176 ceph-mon[25776]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:24.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:24 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/313648142' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/3203283767"}]: dispatch 2024-04-15T22:25:25.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:25 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/313648142' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/3203283767"}]': finished 2024-04-15T22:25:25.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:25 smithi092 ceph-mon[24813]: osdmap e91: 8 total, 8 up, 8 in 2024-04-15T22:25:25.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:25 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3906372013' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/459505342"}]: dispatch 2024-04-15T22:25:25.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:25 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/459505342"}]: dispatch 2024-04-15T22:25:25.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:25 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/313648142' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/3203283767"}]': finished 2024-04-15T22:25:25.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:25 smithi092 ceph-mon[29727]: osdmap e91: 8 total, 8 up, 8 in 2024-04-15T22:25:25.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:25 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3906372013' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/459505342"}]: dispatch 2024-04-15T22:25:25.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:25 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/459505342"}]: dispatch 2024-04-15T22:25:25.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:25 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/313648142' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:6801/3203283767"}]': finished 2024-04-15T22:25:25.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:25 smithi176 ceph-mon[25776]: osdmap e91: 8 total, 8 up, 8 in 2024-04-15T22:25:25.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:25 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3906372013' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/459505342"}]: dispatch 2024-04-15T22:25:25.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:25 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/459505342"}]: dispatch 2024-04-15T22:25:26.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:26 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/459505342"}]': finished 2024-04-15T22:25:26.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:26 smithi092 ceph-mon[29727]: osdmap e92: 8 total, 8 up, 8 in 2024-04-15T22:25:26.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:26 smithi092 ceph-mon[29727]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:26.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:26 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/4104191428' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1778304426"}]: dispatch 2024-04-15T22:25:26.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:26 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1778304426"}]: dispatch 2024-04-15T22:25:26.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:26 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/459505342"}]': finished 2024-04-15T22:25:26.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:26 smithi092 ceph-mon[24813]: osdmap e92: 8 total, 8 up, 8 in 2024-04-15T22:25:26.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:26 smithi092 ceph-mon[24813]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:26.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:26 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/4104191428' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1778304426"}]: dispatch 2024-04-15T22:25:26.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:26 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1778304426"}]: dispatch 2024-04-15T22:25:26.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:26 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/459505342"}]': finished 2024-04-15T22:25:26.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:26 smithi176 ceph-mon[25776]: osdmap e92: 8 total, 8 up, 8 in 2024-04-15T22:25:26.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:26 smithi176 ceph-mon[25776]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:26.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:26 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/4104191428' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1778304426"}]: dispatch 2024-04-15T22:25:26.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:26 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1778304426"}]: dispatch 2024-04-15T22:25:27.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:27 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1778304426"}]': finished 2024-04-15T22:25:27.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:27 smithi176 ceph-mon[25776]: osdmap e93: 8 total, 8 up, 8 in 2024-04-15T22:25:27.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:27 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/570404088' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3495861683"}]: dispatch 2024-04-15T22:25:27.612 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:27 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3495861683"}]: dispatch 2024-04-15T22:25:27.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:27 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1778304426"}]': finished 2024-04-15T22:25:27.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:27 smithi092 ceph-mon[29727]: osdmap e93: 8 total, 8 up, 8 in 2024-04-15T22:25:27.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:27 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/570404088' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3495861683"}]: dispatch 2024-04-15T22:25:27.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:27 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3495861683"}]: dispatch 2024-04-15T22:25:27.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:27 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/1778304426"}]': finished 2024-04-15T22:25:27.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:27 smithi092 ceph-mon[24813]: osdmap e93: 8 total, 8 up, 8 in 2024-04-15T22:25:27.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:27 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/570404088' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3495861683"}]: dispatch 2024-04-15T22:25:27.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:27 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3495861683"}]: dispatch 2024-04-15T22:25:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:28 smithi092 ceph-mon[29727]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3495861683"}]': finished 2024-04-15T22:25:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:28 smithi092 ceph-mon[29727]: osdmap e94: 8 total, 8 up, 8 in 2024-04-15T22:25:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:28 smithi092 ceph-mon[29727]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:28 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:28.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:28 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:28.615 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:28 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:25:28] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:28.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:28 smithi092 ceph-mon[24813]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3495861683"}]': finished 2024-04-15T22:25:28.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:28 smithi092 ceph-mon[24813]: osdmap e94: 8 total, 8 up, 8 in 2024-04-15T22:25:28.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:28 smithi092 ceph-mon[24813]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:28.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:28 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:28.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:28 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:28.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:28 smithi176 ceph-mon[25776]: from='client.? ' entity='client.iscsi.foo.smithi092.hsbujv' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.92:0/3495861683"}]': finished 2024-04-15T22:25:28.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:28 smithi176 ceph-mon[25776]: osdmap e94: 8 total, 8 up, 8 in 2024-04-15T22:25:28.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:28 smithi176 ceph-mon[25776]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:28.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:28 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:28.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:28 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:30.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:30 smithi092 ceph-mon[29727]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1001 B/s rd, 0 op/s 2024-04-15T22:25:30.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:30 smithi092 ceph-mon[24813]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1001 B/s rd, 0 op/s 2024-04-15T22:25:30.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:30 smithi176 ceph-mon[25776]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1001 B/s rd, 0 op/s 2024-04-15T22:25:30.851 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:30 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:25:30] "GET /metrics HTTP/1.1" 200 215035 "" "Prometheus/2.33.4" 2024-04-15T22:25:31.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:31 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:25:31.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:31 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:25:31.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:31 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:25:32.395 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:32 smithi092 ceph-mon[24813]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-04-15T22:25:32.395 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:32 smithi092 ceph-mon[29727]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-04-15T22:25:32.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:32 smithi176 ceph-mon[25776]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 838 B/s rd, 0 op/s 2024-04-15T22:25:33.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:33 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:25:33.509Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:25:33.964 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:33 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:25:33.511Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:25:33.964 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:33 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:25:33.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:25:34.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:25:33 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:25:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:25:34.818 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:34 smithi092 ceph-mon[24813]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:34.818 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:34 smithi092 ceph-mon[29727]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:34.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:34 smithi176 ceph-mon[25776]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:36.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:36 smithi092 ceph-mon[29727]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-15T22:25:36.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:36 smithi092 ceph-mon[24813]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-15T22:25:36.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:36 smithi176 ceph-mon[25776]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-15T22:25:38.463 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:38 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:25:38] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:38.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:38 smithi092 ceph-mon[29727]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-04-15T22:25:38.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:38.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:38.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:38 smithi092 ceph-mon[24813]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-04-15T22:25:38.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:38.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:38.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:38 smithi176 ceph-mon[25776]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1014 B/s rd, 0 op/s 2024-04-15T22:25:38.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:38.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:40.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:40 smithi092 ceph-mon[29727]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:25:40.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:40 smithi092 ceph-mon[24813]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:25:40.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:40 smithi176 ceph-mon[25776]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:25:40.858 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:25:40] "GET /metrics HTTP/1.1" 200 215032 "" "Prometheus/2.33.4" 2024-04-15T22:25:41.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:41 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:25:41.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:41 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:25:41.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:41 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:25:42.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:42 smithi092 ceph-mon[29727]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:25:42.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:42 smithi092 ceph-mon[24813]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:25:42.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:42 smithi176 ceph-mon[25776]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:25:43.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:25:43.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:25:43.964 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:25:43.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:25:43.964 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:25:43.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:25:44.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:25:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:25:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:25:44.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:44 smithi092 ceph-mon[24813]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:44.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:44 smithi092 ceph-mon[29727]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:44.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:44 smithi176 ceph-mon[25776]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:25:46.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:46 smithi176 ceph-mon[25776]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:25:46.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-04-15T22:25:46.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-15T22:25:46.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-04-15T22:25:46.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-15T22:25:46.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-15T22:25:46.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-15T22:25:46.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-15T22:25:46.794 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-15T22:25:46.795 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:25:46.795 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:25:46.795 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:25:46.795 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:25:46.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[29727]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:25:46.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-04-15T22:25:46.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-15T22:25:46.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-04-15T22:25:46.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-15T22:25:46.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-15T22:25:46.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-15T22:25:46.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-15T22:25:46.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-15T22:25:46.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:25:46.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:25:46.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:25:46.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:25:46.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[24813]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:25:46.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-04-15T22:25:46.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-15T22:25:46.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-04-15T22:25:46.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-15T22:25:46.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-15T22:25:46.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-15T22:25:46.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-15T22:25:46.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-15T22:25:46.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:25:46.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:25:46.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:25:46.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:25:47.811 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-04-15T22:25:47.812 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-15T22:25:47.812 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-15T22:25:47.812 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:47 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-04-15T22:25:47.812 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:47 smithi176 ceph-mon[25776]: osdmap e95: 8 total, 8 up, 8 in 2024-04-15T22:25:47.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-04-15T22:25:47.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-15T22:25:47.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-15T22:25:47.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:47 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-04-15T22:25:47.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:47 smithi092 ceph-mon[29727]: osdmap e95: 8 total, 8 up, 8 in 2024-04-15T22:25:47.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-04-15T22:25:47.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-15T22:25:47.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-15T22:25:47.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:47 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-04-15T22:25:47.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:47 smithi092 ceph-mon[24813]: osdmap e95: 8 total, 8 up, 8 in 2024-04-15T22:25:48.629 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:48 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:25:48] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:48.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:48 smithi092 ceph-mon[24813]: pgmap v133: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:48.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:48 smithi092 ceph-mon[24813]: osdmap e96: 8 total, 8 up, 8 in 2024-04-15T22:25:48.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:48 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:48.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:48 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:48.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:48 smithi092 ceph-mon[29727]: pgmap v133: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:48.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:48 smithi092 ceph-mon[29727]: osdmap e96: 8 total, 8 up, 8 in 2024-04-15T22:25:48.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:48 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:48.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:48 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:48.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:48 smithi176 ceph-mon[25776]: pgmap v133: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:48.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:48 smithi176 ceph-mon[25776]: osdmap e96: 8 total, 8 up, 8 in 2024-04-15T22:25:48.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:48 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:48.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:48 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:50.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:50 smithi092 ceph-mon[29727]: pgmap v135: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:50.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:50 smithi092 ceph-mon[24813]: pgmap v135: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:50.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:50 smithi176 ceph-mon[25776]: pgmap v135: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:50.869 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:25:50] "GET /metrics HTTP/1.1" 200 214958 "" "Prometheus/2.33.4" 2024-04-15T22:25:51.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:51 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:25:51.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:51 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:25:51.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:51 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:25:52.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:52 smithi092 ceph-mon[24813]: pgmap v136: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:52.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:52 smithi092 ceph-mon[29727]: pgmap v136: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:52.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:52 smithi176 ceph-mon[25776]: pgmap v136: 161 pgs: 4 peering, 157 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail 2024-04-15T22:25:53.713 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:25:53.510Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:25:53.714 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:25:53.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:25:53.714 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:25:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:25:53.512Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:25:54.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:25:53 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:25:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:25:54.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:54 smithi092 ceph-mon[29727]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 50 B/s, 0 objects/s recovering 2024-04-15T22:25:54.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:54 smithi092 ceph-mon[24813]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 50 B/s, 0 objects/s recovering 2024-04-15T22:25:54.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:54 smithi176 ceph-mon[25776]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 50 B/s, 0 objects/s recovering 2024-04-15T22:25:56.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:56 smithi092 ceph-mon[24813]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 B/s, 0 objects/s recovering 2024-04-15T22:25:56.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:56 smithi092 ceph-mon[29727]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 B/s, 0 objects/s recovering 2024-04-15T22:25:56.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:56 smithi176 ceph-mon[25776]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 45 B/s, 0 objects/s recovering 2024-04-15T22:25:58.631 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:25:58 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:25:58] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:25:58.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:58 smithi092 ceph-mon[24813]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 307 B/s wr, 1 op/s; 40 B/s, 0 objects/s recovering 2024-04-15T22:25:58.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:58 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:58.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:25:58 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:58.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:58 smithi092 ceph-mon[29727]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 307 B/s wr, 1 op/s; 40 B/s, 0 objects/s recovering 2024-04-15T22:25:58.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:58 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:58.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:25:58 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:58.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:58 smithi176 ceph-mon[25776]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 307 B/s wr, 1 op/s; 40 B/s, 0 objects/s recovering 2024-04-15T22:25:58.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:58 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:25:58.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:25:58 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:00.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:00 smithi092 ceph-mon[24813]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 257 B/s wr, 1 op/s; 33 B/s, 0 objects/s recovering 2024-04-15T22:26:00.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:00 smithi092 ceph-mon[29727]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 257 B/s wr, 1 op/s; 33 B/s, 0 objects/s recovering 2024-04-15T22:26:00.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:00 smithi176 ceph-mon[25776]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 257 B/s wr, 1 op/s; 33 B/s, 0 objects/s recovering 2024-04-15T22:26:00.880 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:26:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:26:00] "GET /metrics HTTP/1.1" 200 214958 "" "Prometheus/2.33.4" 2024-04-15T22:26:01.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:01 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:01.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:01 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:01.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:01 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:02.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:02 smithi092 ceph-mon[24813]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 255 B/s wr, 1 op/s; 33 B/s, 0 objects/s recovering 2024-04-15T22:26:02.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:02 smithi092 ceph-mon[29727]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 255 B/s wr, 1 op/s; 33 B/s, 0 objects/s recovering 2024-04-15T22:26:02.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:02 smithi176 ceph-mon[25776]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 255 B/s wr, 1 op/s; 33 B/s, 0 objects/s recovering 2024-04-15T22:26:03.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:26:03.511Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:26:03.964 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:26:03.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:26:03.964 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:26:03.513Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:26:04.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:26:03 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:26:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:26:04.944 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:04 smithi176 ceph-mon[25776]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 33 B/s, 0 objects/s recovering 2024-04-15T22:26:04.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:04 smithi092 ceph-mon[24813]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 33 B/s, 0 objects/s recovering 2024-04-15T22:26:04.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:04 smithi092 ceph-mon[29727]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s; 33 B/s, 0 objects/s recovering 2024-04-15T22:26:06.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:06 smithi092 ceph-mon[24813]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:06.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:06 smithi092 ceph-mon[29727]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:06.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:06 smithi176 ceph-mon[25776]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:08.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:08 smithi092 ceph-mon[24813]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:08.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:08 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:08.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:08 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:08.714 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:26:08 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:26:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:26:08.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:08 smithi092 ceph-mon[29727]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:08.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:08 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:08.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:08 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:08.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:08 smithi176 ceph-mon[25776]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:08.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:08 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:08.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:08 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:10.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:10 smithi092 ceph-mon[29727]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:10.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:10 smithi092 ceph-mon[24813]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:10.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:10 smithi176 ceph-mon[25776]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:10.890 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:26:10 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:26:10] "GET /metrics HTTP/1.1" 200 214969 "" "Prometheus/2.33.4" 2024-04-15T22:26:11.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:11 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:11.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:11 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:11.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:11 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:12.296 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:12 smithi092 ceph-mon[29727]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:12.297 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:12 smithi092 ceph-mon[24813]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:12.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:12 smithi176 ceph-mon[25776]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:13.564 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:26:13.512Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:26:13.564 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:26:13.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:26:13.564 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:26:13.514Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:26:14.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:26:13 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:26:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:26:14.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:14 smithi092 ceph-mon[29727]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:14.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:14 smithi092 ceph-mon[24813]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:14.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:14 smithi176 ceph-mon[25776]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:16.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:16 smithi092 ceph-mon[24813]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:16.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:16 smithi092 ceph-mon[29727]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:16.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:16 smithi176 ceph-mon[25776]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:18.628 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:26:18] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:26:18.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[24813]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:18.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:26:18.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:26:18.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:26:18.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:18.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:18.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:18.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[29727]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:18.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:26:18.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:26:18.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:26:18.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:18.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:18.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:18 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:18.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:18 smithi176 ceph-mon[25776]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:18.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:18 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:26:18.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:18 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:26:18.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:18 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:26:18.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:18 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:18.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:18 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:18.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:18 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:20.466 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:20 smithi092 ceph-mon[24813]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:20.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:20 smithi092 ceph-mon[29727]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:20.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:20 smithi176 ceph-mon[25776]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:20.897 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:26:20 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:26:20] "GET /metrics HTTP/1.1" 200 214950 "" "Prometheus/2.33.4" 2024-04-15T22:26:21.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:21 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:21.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:21 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:21.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:21 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:22.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:22 smithi092 ceph-mon[29727]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:22.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:22 smithi092 ceph-mon[24813]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:22.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:22 smithi176 ceph-mon[25776]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:23.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:23 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:26:23.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:26:23.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:23 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:26:23.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:26:23.964 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:23 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:26:23.515Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:26:24.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:26:23 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:26:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:26:24.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:24 smithi092 ceph-mon[29727]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:24.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:24 smithi092 ceph-mon[24813]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:24.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:24 smithi176 ceph-mon[25776]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:26.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:26 smithi092 ceph-mon[29727]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:26.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:26 smithi092 ceph-mon[24813]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:26.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:26 smithi176 ceph-mon[25776]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:28.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:28 smithi092 ceph-mon[29727]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:28.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:28 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:28.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:28 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:28.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:28 smithi092 ceph-mon[24813]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:28.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:28 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:28.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:28 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:28.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:26:28 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:26:28] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:26:28.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:28 smithi176 ceph-mon[25776]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:28.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:28 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:28.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:28 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:30.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:30 smithi092 ceph-mon[24813]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-15T22:26:30.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:30 smithi092 ceph-mon[29727]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-15T22:26:30.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:30 smithi176 ceph-mon[25776]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-15T22:26:30.902 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:26:30 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:26:30] "GET /metrics HTTP/1.1" 200 214950 "" "Prometheus/2.33.4" 2024-04-15T22:26:31.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:31 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:31.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:31 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:31.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:31 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:32.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:32 smithi092 ceph-mon[29727]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-15T22:26:32.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:32 smithi092 ceph-mon[24813]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-15T22:26:32.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:32 smithi176 ceph-mon[25776]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-15T22:26:33.714 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:33 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:26:33.513Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:26:33.714 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:33 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:26:33.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:26:33.714 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:33 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:26:33.516Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:26:34.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:26:33 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:26:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:26:34.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:34 smithi092 ceph-mon[24813]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:34.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:34 smithi092 ceph-mon[29727]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:34.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:34 smithi176 ceph-mon[25776]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:36.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:36 smithi092 ceph-mon[24813]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-15T22:26:36.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:36 smithi092 ceph-mon[29727]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-15T22:26:36.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:36 smithi176 ceph-mon[25776]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-04-15T22:26:38.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:38 smithi092 ceph-mon[24813]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:38.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:38.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:38 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:38.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:26:38 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:26:38] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:26:38.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:38 smithi092 ceph-mon[29727]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:38.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:38.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:38 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:38.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:38 smithi176 ceph-mon[25776]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:38.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:38.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:38 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:40.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:40 smithi092 ceph-mon[24813]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:40.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:40 smithi092 ceph-mon[29727]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:40.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:40 smithi176 ceph-mon[25776]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:40.903 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:26:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:26:40] "GET /metrics HTTP/1.1" 200 214948 "" "Prometheus/2.33.4" 2024-04-15T22:26:41.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:41 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:41.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:41 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:41.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:41 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:42.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:42 smithi092 ceph-mon[24813]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:42.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:42 smithi092 ceph-mon[29727]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:42.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:42 smithi176 ceph-mon[25776]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:43.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:26:43.515Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:26:43.964 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:26:43.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:26:43.964 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:26:43.517Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:26:44.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:26:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:26:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:26:44.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:44 smithi092 ceph-mon[24813]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:44.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:44 smithi092 ceph-mon[29727]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:44.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:44 smithi176 ceph-mon[25776]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:46.695 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:46 smithi176 ceph-mon[25776]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:46.696 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:26:46.696 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:26:46.696 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:26:46.696 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:26:46.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:46 smithi092 ceph-mon[29727]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:46.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:26:46.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:26:46.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:26:46.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:26:46.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:46 smithi092 ceph-mon[24813]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:46.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:26:46.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:26:46.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:26:46.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:26:48.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:48 smithi092 ceph-mon[29727]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:48.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:48 smithi092 ceph-mon[24813]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:48.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:26:48 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:26:48] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:26:48.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:48 smithi176 ceph-mon[25776]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:49.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:49 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:49.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:49 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:49.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:49 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:49.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:49 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:49.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:49 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:49.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:49 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:50.913 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:50 smithi092 ceph-mon[29727]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:50.913 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:50 smithi092 ceph-mon[24813]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:50.913 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:26:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:26:50] "GET /metrics HTTP/1.1" 200 214959 "" "Prometheus/2.33.4" 2024-04-15T22:26:50.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:50 smithi176 ceph-mon[25776]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:51.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:51 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:51.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:51 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:51.977 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:51 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:26:52.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:52 smithi092 ceph-mon[24813]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:52.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:52 smithi092 ceph-mon[29727]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:52.977 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:52 smithi176 ceph-mon[25776]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:53.713 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:26:53.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:26:53.713 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:26:53.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:26:53.714 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:26:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:26:53.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:26:54.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:26:53 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:26:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:26:54.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:54 smithi092 ceph-mon[29727]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:54.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:54 smithi092 ceph-mon[24813]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:54.977 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:54 smithi176 ceph-mon[25776]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:56.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:56 smithi092 ceph-mon[29727]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:56.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:56 smithi092 ceph-mon[24813]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:56.977 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:56 smithi176 ceph-mon[25776]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:26:58.963 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:26:58 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:26:58] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:26:58.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:58 smithi092 ceph-mon[24813]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:58.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:58 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:58.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:26:58 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:58.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:58 smithi092 ceph-mon[29727]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:58.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:58 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:58.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:26:58 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:58.977 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:58 smithi176 ceph-mon[25776]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:26:58.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:58 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:58.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:26:58 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:26:59.530 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-15T22:27:00.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:00 smithi092 ceph-mon[29727]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:00.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:00 smithi092 ceph-mon[24813]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:00.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:00 smithi176 ceph-mon[25776]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:00.766 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:27:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:27:00] "GET /metrics HTTP/1.1" 200 214959 "" "Prometheus/2.33.4" 2024-04-15T22:27:01.457 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:01 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:01.457 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:01 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:01.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:01 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:02.123 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-15T22:27:02.438 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:02 smithi092 ceph-mon[24813]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:02.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:02 smithi092 ceph-mon[29727]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:02.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:02 smithi176 ceph-mon[25776]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:03.644 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:27:03.516Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:27:03.644 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:27:03.518Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:27:03.644 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:27:03.519Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:27:04.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:27:03 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:27:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:27:04.664 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-15T22:27:04.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:04 smithi092 ceph-mon[24813]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:04.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:04 smithi092 ceph-mon[29727]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:04.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:04 smithi176 ceph-mon[25776]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:06.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:06 smithi176 ceph-mon[25776]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:06.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:06 smithi092 ceph-mon[24813]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:06.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:06 smithi092 ceph-mon[29727]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:07.386 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-15T22:27:08.772 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:08 smithi176 ceph-mon[25776]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:08.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:08 smithi092 ceph-mon[24813]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:08.964 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:27:08 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:27:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:27:08.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:08 smithi092 ceph-mon[29727]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:09.433 INFO:teuthology.orchestra.run.smithi092.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:27:10.000 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:10.000 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:10.000 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:10.000 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:27:10.000 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:27:10.000 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:27:10.000 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:10.000 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:10.000 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:10.000 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:10.001 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:27:10.001 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:27:10.001 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:27:10.001 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:09 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:10.023 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T22:27:10.034 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi092.front.sepia.ceph.com 2024-04-15T22:27:10.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-15T22:27:10.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:09 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:10.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:09 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:10.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:09 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:10.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:09 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:27:10.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:09 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:27:10.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:09 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:27:10.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:09 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:10.373 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:27:10.244+0000 7f660912d700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T22:27:10.713 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:27:10] "GET /metrics HTTP/1.1" 200 214969 "" "Prometheus/2.33.4" 2024-04-15T22:27:11.039 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[29727]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:11.040 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[29727]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:27:11.040 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[29727]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:27:11.040 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[29727]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:11.040 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:11.040 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:27:11.040 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:27:11.040 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:27:11.040 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:11.040 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[29727]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-15T22:27:11.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[24813]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:11.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[24813]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:27:11.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[24813]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:27:11.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[24813]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:11.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:11.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:27:11.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:27:11.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:27:11.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:11.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:10 smithi092 ceph-mon[24813]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-15T22:27:11.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:10 smithi176 ceph-mon[25776]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:11.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:10 smithi176 ceph-mon[25776]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:27:11.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:10 smithi176 ceph-mon[25776]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:27:11.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:10 smithi176 ceph-mon[25776]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:11.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:10 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:11.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:10 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:27:11.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:10 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:27:11.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:10 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:27:11.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:10 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:11.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:10 smithi176 ceph-mon[25776]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-15T22:27:11.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:11 smithi092 ceph-mon[29727]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T22:27:11.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:11 smithi092 ceph-mon[29727]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:11.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:11 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:11.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:11 smithi092 ceph-mon[24813]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T22:27:11.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:11 smithi092 ceph-mon[24813]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:11.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:11 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:12.013 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-04-15T22:27:12.150 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:11 smithi176 ceph-mon[25776]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T22:27:12.150 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:11 smithi176 ceph-mon[25776]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:12.150 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:11 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:12.320 INFO:teuthology.orchestra.run.smithi092.stdout:"Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-15T22:27:12.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:12 smithi092 ceph-mon[24813]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:12.872 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:12 smithi092 ceph-mon[29727]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:12.896 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch ps' 2024-04-15T22:27:12.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:12 smithi176 ceph-mon[25776]: from='client.24970 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:13.775 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:27:13.518Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:27:13.775 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:27:13.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:27:13.775 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:27:13.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:27:14.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:13 smithi092 ceph-mon[24813]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-15T22:27:14.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:13 smithi092 ceph-mon[24813]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:14.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:13 smithi092 ceph-mon[29727]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-15T22:27:14.030 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:13 smithi092 ceph-mon[29727]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:14.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:13 smithi176 ceph-mon[25776]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-04-15T22:27:14.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:13 smithi176 ceph-mon[25776]: from='client.24976 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:14.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:27:13 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:27:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:27:15.436 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:15 smithi092 ceph-mon[29727]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:15.436 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:15 smithi092 ceph-mon[24813]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:15.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:15 smithi176 ceph-mon[25776]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:16.412 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-15T22:27:16.412 INFO:teuthology.orchestra.run.smithi092.stdout:agent.smithi092 smithi092 running 17s ago 8m - - 2024-04-15T22:27:16.412 INFO:teuthology.orchestra.run.smithi092.stdout:agent.smithi176 smithi176 running 7s ago 8m - - 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (2m) 17s ago 3m 27.4M - ba2b418f427c 5d76ee1dbbdb 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi176 *:3000 running (2m) 7s ago 2m 54.9M - 8.3.5 dad864ee21e9 1233e94881e9 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.hsbujv smithi092 running (2m) 17s ago 2m 48.5M - 3.5 e1d6a67b021e 75352e4c4f7e 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi176 *:8443 running (8m) 7s ago 8m 412M - 17.2.0 e1d6a67b021e 6460f0b4f850 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:9283 running (9m) 17s ago 9m 464M - 17.2.0 e1d6a67b021e 4e75752c42e9 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (9m) 17s ago 9m 50.3M 2048M 17.2.0 e1d6a67b021e 5e439f1bb752 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi176 running (8m) 7s ago 8m 41.6M 2048M 17.2.0 e1d6a67b021e a93cba3b76ed 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (8m) 17s ago 8m 40.4M 2048M 17.2.0 e1d6a67b021e 69aa2c1b085d 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (4m) 17s ago 4m 17.4M - 1dbe0e931976 2b352622d41e 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi176 *:9100 running (4m) 7s ago 4m 16.8M - 1dbe0e931976 774c4598b14f 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (7m) 17s ago 7m 52.3M 4096M 17.2.0 e1d6a67b021e 77d7f9c5f13e 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (7m) 17s ago 7m 51.2M 4096M 17.2.0 e1d6a67b021e c1a9a397f575 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (6m) 17s ago 6m 46.0M 4096M 17.2.0 e1d6a67b021e 7e162849c1f7 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (6m) 17s ago 6m 48.6M 4096M 17.2.0 e1d6a67b021e c26f54e54812 2024-04-15T22:27:16.413 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi176 running (6m) 7s ago 6m 51.5M 4096M 17.2.0 e1d6a67b021e 300e4da7c469 2024-04-15T22:27:16.414 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi176 running (5m) 7s ago 5m 47.7M 4096M 17.2.0 e1d6a67b021e 9ba0015eb74a 2024-04-15T22:27:16.414 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi176 running (5m) 7s ago 5m 45.9M 4096M 17.2.0 e1d6a67b021e 1727b9b75635 2024-04-15T22:27:16.414 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi176 running (4m) 7s ago 4m 48.8M 4096M 17.2.0 e1d6a67b021e 1ba050fa6fb8 2024-04-15T22:27:16.414 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi176 *:9095 running (2m) 7s ago 4m 62.8M - 514e6a882f6e 4c2f0c2b64ee 2024-04-15T22:27:16.414 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.xmypxq smithi092 *:8000 running (2m) 17s ago 2m 69.3M - 17.2.0 e1d6a67b021e 5252a0607e85 2024-04-15T22:27:16.414 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi176.twznrb smithi176 *:8000 running (2m) 7s ago 2m 71.9M - 17.2.0 e1d6a67b021e 7174181e64a6 2024-04-15T22:27:16.414 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.juhhhh smithi092 *:80 running (2m) 17s ago 2m 71.7M - 17.2.0 e1d6a67b021e 0e5af7d40780 2024-04-15T22:27:16.414 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi176.fwngry smithi176 *:80 running (2m) 7s ago 2m 68.7M - 17.2.0 e1d6a67b021e d8bd6332814f 2024-04-15T22:27:17.194 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:17 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:17.194 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:17 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:17.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:17 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:17.703 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph versions' 2024-04-15T22:27:18.321 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:18 smithi092 ceph-mon[29727]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:18.321 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:18 smithi092 ceph-mon[29727]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:18.321 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:18 smithi092 ceph-mon[24813]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:18.321 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:18 smithi092 ceph-mon[24813]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:18.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:18 smithi176 ceph-mon[25776]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:18.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:18 smithi176 ceph-mon[25776]: from='client.15081 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:27:18.963 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:27:18 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:27:18] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:27:19.693 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-15T22:27:19.693 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-04-15T22:27:19.693 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-15T22:27:19.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:27:19.693 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-04-15T22:27:19.693 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-15T22:27:19.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:27:19.693 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-04-15T22:27:19.693 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-15T22:27:19.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:27:19.694 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": {}, 2024-04-15T22:27:19.694 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-04-15T22:27:19.694 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-15T22:27:19.694 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:27:19.694 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-04-15T22:27:19.694 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-15T22:27:19.694 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-15T22:27:19.694 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-15T22:27:19.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:19 smithi092 ceph-mon[24813]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:19.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:19 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:19.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:19 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:19.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:19 smithi092 ceph-mon[29727]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:19.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:19 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:19.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:19 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:19.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:19 smithi176 ceph-mon[25776]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:19.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:19 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:19.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:19 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:20.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-15T22:27:20.892 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:20 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2536905297' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:27:20.892 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:20 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2536905297' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:27:20.892 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:27:20 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:27:20] "GET /metrics HTTP/1.1" 200 215024 "" "Prometheus/2.33.4" 2024-04-15T22:27:20.977 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:20 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2536905297' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:27:21.766 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:21 smithi092 ceph-mon[29727]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:21.766 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:21 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:21.767 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:21 smithi092 ceph-mon[24813]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:21.767 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:21 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:21.965 INFO:teuthology.orchestra.run.smithi092.stdout:wait for servicemap items w/ changing names to refresh 2024-04-15T22:27:21.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:21 smithi176 ceph-mon[25776]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 959 B/s rd, 0 op/s 2024-04-15T22:27:21.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:21 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:22.646 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'sleep 60' 2024-04-15T22:27:23.651 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:23 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:27:23.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:27:23.651 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:23 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:27:23.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:27:23.651 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:23 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:27:23.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:27:23.942 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:23 smithi092 ceph-mon[24813]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:23.942 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:23 smithi092 ceph-mon[29727]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:23.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:23 smithi176 ceph-mon[25776]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:23.978 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:27:23 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:27:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:27:25.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:25 smithi092 ceph-mon[29727]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:25.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:25 smithi092 ceph-mon[24813]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:25.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:25 smithi176 ceph-mon[25776]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:27.453 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:27 smithi176 ceph-mon[25776]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:27.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:27 smithi092 ceph-mon[29727]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:27.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:27 smithi092 ceph-mon[24813]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:29.213 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:27:28 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:27:28] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:27:30.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:29 smithi092 ceph-mon[29727]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:30.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:29 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:30.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:29 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:30.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:29 smithi092 ceph-mon[24813]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:30.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:29 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:30.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:29 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:30.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:29 smithi176 ceph-mon[25776]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:30.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:29 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:30.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:29 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:30.942 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:27:30 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:27:30] "GET /metrics HTTP/1.1" 200 215024 "" "Prometheus/2.33.4" 2024-04-15T22:27:31.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:31 smithi092 ceph-mon[24813]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:31.963 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:31 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:31.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:31 smithi092 ceph-mon[29727]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:31.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:31 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:32.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:31 smithi176 ceph-mon[25776]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:32.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:31 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:33.870 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:33 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:27:33.519Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:27:33.871 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:33 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:27:33.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:27:33.871 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:33 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:27:33.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:27:34.187 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:33 smithi176 ceph-mon[25776]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:34.188 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:27:33 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:27:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:27:34.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:33 smithi092 ceph-mon[29727]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:34.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:33 smithi092 ceph-mon[24813]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:35.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:35 smithi092 ceph-mon[29727]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:35.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:35 smithi092 ceph-mon[24813]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:35.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:35 smithi176 ceph-mon[25776]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:37.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:37 smithi092 ceph-mon[24813]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:37.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:37 smithi092 ceph-mon[29727]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:37.727 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:37 smithi176 ceph-mon[25776]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:39.213 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:27:38 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:27:38] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:27:40.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:39 smithi092 ceph-mon[29727]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:40.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:39 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:40.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:39 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:40.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:39 smithi092 ceph-mon[24813]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:40.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:39 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:40.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:39 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:40.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:39 smithi176 ceph-mon[25776]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:40.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:39 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:40.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:39 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:40.953 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:27:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:27:40] "GET /metrics HTTP/1.1" 200 215007 "" "Prometheus/2.33.4" 2024-04-15T22:27:42.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:41 smithi092 ceph-mon[24813]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:42.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:41 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:42.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:41 smithi092 ceph-mon[29727]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:42.214 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:41 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:42.227 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:41 smithi176 ceph-mon[25776]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:42.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:41 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:43.794 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:27:43.520Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:27:43.794 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:27:43.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:27:43.794 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:43 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:27:43.522Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:27:43.978 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:43 smithi176 ceph-mon[25776]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:43.978 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:27:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:27:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:27:44.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:43 smithi092 ceph-mon[24813]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:44.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:43 smithi092 ceph-mon[29727]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:45.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:45 smithi092 ceph-mon[29727]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:45.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:45 smithi092 ceph-mon[24813]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:45.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:45 smithi176 ceph-mon[25776]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:46.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:27:46.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:27:46.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:46 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:27:46.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:46 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:27:46.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:27:46.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:27:46.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:46 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:27:46.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:46 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:27:46.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:27:46.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T22:27:46.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:46 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:27:46.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:46 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T22:27:47.444 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:47 smithi176 ceph-mon[25776]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:47.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:47 smithi092 ceph-mon[29727]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:47.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:47 smithi092 ceph-mon[24813]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:49.463 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:27:48 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:27:48] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:27:50.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:49 smithi176 ceph-mon[25776]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:50.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:49 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:50.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:49 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:50.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:49 smithi092 ceph-mon[29727]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:50.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:49 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:50.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:49 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:50.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:49 smithi092 ceph-mon[24813]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:50.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:49 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:50.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:49 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:27:50.962 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:27:50 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:27:50] "GET /metrics HTTP/1.1" 200 214996 "" "Prometheus/2.33.4" 2024-04-15T22:27:52.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:51 smithi092 ceph-mon[29727]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:52.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:51 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:52.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:51 smithi092 ceph-mon[24813]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:52.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:51 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:52.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:51 smithi176 ceph-mon[25776]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:52.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:51 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:27:53.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:27:53.521Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:27:53.964 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:27:53.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:27:53.964 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:27:53 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:27:53.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:27:54.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:53 smithi176 ceph-mon[25776]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:54.229 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:27:53 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:27:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:27:54.389 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:53 smithi092 ceph-mon[24813]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:54.389 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:53 smithi092 ceph-mon[29727]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:27:55.446 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:55 smithi092 ceph-mon[24813]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:55.446 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:55 smithi092 ceph-mon[29727]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:55.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:55 smithi176 ceph-mon[25776]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:57.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:57 smithi092 ceph-mon[24813]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:57.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:57 smithi092 ceph-mon[29727]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:57.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:57 smithi176 ceph-mon[25776]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:27:59.213 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:27:58 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:27:58] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:28:00.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:59 smithi092 ceph-mon[29727]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:00.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:59 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:00.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:27:59 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:00.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:59 smithi092 ceph-mon[24813]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:00.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:59 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:00.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:27:59 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:00.227 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:59 smithi176 ceph-mon[25776]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:00.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:59 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:00.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:27:59 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:00.963 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:28:00] "GET /metrics HTTP/1.1" 200 214996 "" "Prometheus/2.33.4" 2024-04-15T22:28:02.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:01 smithi092 ceph-mon[29727]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:02.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:01 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:02.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:01 smithi092 ceph-mon[24813]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:02.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:01 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:02.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:01 smithi176 ceph-mon[25776]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:02.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:01 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:03.909 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:28:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:28:03.521Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:28:03.909 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:28:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:28:03.524Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:28:03.909 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:28:03 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:28:03.524Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:28:04.213 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:03 smithi092 ceph-mon[29727]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:04.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:03 smithi092 ceph-mon[24813]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:04.227 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:03 smithi176 ceph-mon[25776]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:04.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:28:03 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:28:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:28:05.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:05 smithi092 ceph-mon[24813]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:05.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:05 smithi092 ceph-mon[29727]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:05.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:05 smithi176 ceph-mon[25776]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:07.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:07 smithi092 ceph-mon[29727]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:07.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:07 smithi092 ceph-mon[24813]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:07.727 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:07 smithi176 ceph-mon[25776]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:09.463 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:09 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:28:09] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:28:10.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:10 smithi176 ceph-mon[25776]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:10.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:10 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:10.229 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:10 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:10.322 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:10 smithi092 ceph-mon[29727]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:10.323 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:10 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:10.323 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:10 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:10.323 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:10 smithi092 ceph-mon[24813]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:10.323 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:10 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:10.323 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:10 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:10.963 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:10 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:28:10] "GET /metrics HTTP/1.1" 200 214968 "" "Prometheus/2.33.4" 2024-04-15T22:28:11.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:11 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:28:11.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:11 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:28:11.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:11 smithi092 ceph-mon[29727]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:28:11.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:11 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:11.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:11 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:11.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:11 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:28:11.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:11 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:28:11.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:11 smithi092 ceph-mon[24813]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:28:11.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:11 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:11.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:11 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:11.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:11 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T22:28:11.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:11 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T22:28:11.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:11 smithi176 ceph-mon[25776]: from='mgr.24340 172.21.15.92:0/4257175553' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T22:28:11.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:11 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:11.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:11 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:12.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:12 smithi092 ceph-mon[24813]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:12.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:12 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:12.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:12 smithi092 ceph-mon[29727]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:12.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:12 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:12.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:12 smithi176 ceph-mon[25776]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:12.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:12 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:13.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:13 smithi176 ceph-mon[25776]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:13.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:13 smithi092 ceph-mon[29727]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:13.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:13 smithi092 ceph-mon[24813]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:13.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:28:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:28:13.522Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:28:13.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:28:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:28:13.523Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:28:13.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:28:13 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:28:13.524Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:28:14.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:28:13 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:28:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:28:15.456 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:15 smithi092 ceph-mon[24813]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:15.456 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:15 smithi092 ceph-mon[29727]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:15.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:15 smithi176 ceph-mon[25776]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:17.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:17 smithi092 ceph-mon[29727]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:17.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:17 smithi092 ceph-mon[24813]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:17.727 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:17 smithi176 ceph-mon[25776]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:19.463 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:19 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:28:19] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:28:20.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:20 smithi092 ceph-mon[24813]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:20.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:20 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:20.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:20 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:20.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:20 smithi092 ceph-mon[29727]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:20.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:20 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:20.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:20 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:20.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:20 smithi176 ceph-mon[25776]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:20.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:20 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:20.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:20 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:20.963 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:20 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:28:20] "GET /metrics HTTP/1.1" 200 214974 "" "Prometheus/2.33.4" 2024-04-15T22:28:22.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:22 smithi092 ceph-mon[29727]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:22.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:22 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:22.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:22 smithi092 ceph-mon[24813]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:22.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:22 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:22.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:22 smithi176 ceph-mon[25776]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:22.478 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:22 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:23.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:28:23 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:28:23.523Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:28:23.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:28:23 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:28:23.525Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:28:23.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:28:23 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=warn ts=2024-04-15T22:28:23.525Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs" 2024-04-15T22:28:24.228 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:24 smithi176 ceph-mon[25776]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:24.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:28:23 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:28:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:28:24.336 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:24 smithi092 ceph-mon[29727]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:24.336 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:24 smithi092 ceph-mon[24813]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:25.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch ps' 2024-04-15T22:28:25.429 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:25 smithi092 ceph-mon[29727]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:25.429 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:25 smithi092 ceph-mon[24813]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:25.481 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:25 smithi176 ceph-mon[25776]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:27.059 INFO:teuthology.orchestra.run.smithi092.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-15T22:28:27.059 INFO:teuthology.orchestra.run.smithi092.stdout:agent.smithi092 smithi092 running 8s ago 10m - - 2024-04-15T22:28:27.059 INFO:teuthology.orchestra.run.smithi092.stdout:agent.smithi176 smithi176 running 17s ago 9m - - 2024-04-15T22:28:27.059 INFO:teuthology.orchestra.run.smithi092.stdout:alertmanager.a smithi092 *:9093,9094 running (3m) 8s ago 4m 27.0M - ba2b418f427c 5d76ee1dbbdb 2024-04-15T22:28:27.059 INFO:teuthology.orchestra.run.smithi092.stdout:grafana.a smithi176 *:3000 running (4m) 17s ago 4m 54.5M - 8.3.5 dad864ee21e9 1233e94881e9 2024-04-15T22:28:27.059 INFO:teuthology.orchestra.run.smithi092.stdout:iscsi.foo.smithi092.hsbujv smithi092 running (3m) 8s ago 3m 50.9M - 3.5 e1d6a67b021e 75352e4c4f7e 2024-04-15T22:28:27.059 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.x smithi176 *:8443 running (9m) 17s ago 9m 412M - 17.2.0 e1d6a67b021e 6460f0b4f850 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:mgr.y smithi092 *:9283 running (10m) 8s ago 10m 467M - 17.2.0 e1d6a67b021e 4e75752c42e9 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a smithi092 running (11m) 8s ago 11m 52.8M 2048M 17.2.0 e1d6a67b021e 5e439f1bb752 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b smithi176 running (9m) 17s ago 9m 38.0M 2048M 17.2.0 e1d6a67b021e a93cba3b76ed 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c smithi092 running (9m) 8s ago 9m 40.9M 2048M 17.2.0 e1d6a67b021e 69aa2c1b085d 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.a smithi092 *:9100 running (5m) 8s ago 5m 17.8M - 1dbe0e931976 2b352622d41e 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:node-exporter.b smithi176 *:9100 running (5m) 17s ago 5m 16.5M - 1dbe0e931976 774c4598b14f 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 smithi092 running (8m) 8s ago 8m 52.9M 4096M 17.2.0 e1d6a67b021e 77d7f9c5f13e 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 smithi092 running (8m) 8s ago 8m 51.9M 4096M 17.2.0 e1d6a67b021e c1a9a397f575 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 smithi092 running (8m) 8s ago 8m 46.5M 4096M 17.2.0 e1d6a67b021e 7e162849c1f7 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 smithi092 running (7m) 8s ago 7m 49.5M 4096M 17.2.0 e1d6a67b021e c26f54e54812 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 smithi176 running (7m) 17s ago 7m 51.9M 4096M 17.2.0 e1d6a67b021e 300e4da7c469 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 smithi176 running (6m) 17s ago 6m 48.0M 4096M 17.2.0 e1d6a67b021e 9ba0015eb74a 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 smithi176 running (6m) 17s ago 6m 46.2M 4096M 17.2.0 e1d6a67b021e 1727b9b75635 2024-04-15T22:28:27.060 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 smithi176 running (6m) 17s ago 6m 49.2M 4096M 17.2.0 e1d6a67b021e 1ba050fa6fb8 2024-04-15T22:28:27.061 INFO:teuthology.orchestra.run.smithi092.stdout:prometheus.a smithi176 *:9095 running (3m) 17s ago 5m 64.9M - 514e6a882f6e 4c2f0c2b64ee 2024-04-15T22:28:27.061 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi092.xmypxq smithi092 *:8000 running (3m) 8s ago 3m 69.8M - 17.2.0 e1d6a67b021e 5252a0607e85 2024-04-15T22:28:27.061 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.foo.smithi176.twznrb smithi176 *:8000 running (3m) 17s ago 3m 72.0M - 17.2.0 e1d6a67b021e 7174181e64a6 2024-04-15T22:28:27.061 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi092.juhhhh smithi092 *:80 running (3m) 8s ago 3m 72.2M - 17.2.0 e1d6a67b021e 0e5af7d40780 2024-04-15T22:28:27.061 INFO:teuthology.orchestra.run.smithi092.stdout:rgw.smpl.smithi176.fwngry smithi176 *:80 running (3m) 17s ago 3m 69.0M - 17.2.0 e1d6a67b021e d8bd6332814f 2024-04-15T22:28:27.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:27 smithi092 ceph-mon[24813]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:27.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:27 smithi092 ceph-mon[24813]: from='client.24994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:28:27.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:27 smithi092 ceph-mon[29727]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:27.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:27 smithi092 ceph-mon[29727]: from='client.24994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:28:27.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:27 smithi176 ceph-mon[25776]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:27.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:27 smithi176 ceph-mon[25776]: from='client.24994 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:28:27.882 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph versions' 2024-04-15T22:28:29.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:29 smithi092 ceph-mon[29727]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:29.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:29 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:29.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:29 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:29.464 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:29 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.176 - - [15/Apr/2024:22:28:29] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:28:29.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:29 smithi092 ceph-mon[24813]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:29.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:29 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:29.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:29 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:29.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:29 smithi176 ceph-mon[25776]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:29.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:29 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:29.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:29 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:30.006 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-15T22:28:30.006 INFO:teuthology.orchestra.run.smithi092.stdout: "mon": { 2024-04-15T22:28:30.006 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-15T22:28:30.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:28:30.006 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": { 2024-04-15T22:28:30.007 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-15T22:28:30.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:28:30.007 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": { 2024-04-15T22:28:30.007 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-15T22:28:30.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:28:30.007 INFO:teuthology.orchestra.run.smithi092.stdout: "mds": {}, 2024-04-15T22:28:30.007 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": { 2024-04-15T22:28:30.007 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-15T22:28:30.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-04-15T22:28:30.007 INFO:teuthology.orchestra.run.smithi092.stdout: "overall": { 2024-04-15T22:28:30.007 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-15T22:28:30.007 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-04-15T22:28:30.007 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-15T22:28:30.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:30 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3320185394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:30.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:30 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3320185394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:30.582 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:30 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3320185394' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:30.701 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch upgrade status' 2024-04-15T22:28:30.963 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:30 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:28:30] "GET /metrics HTTP/1.1" 200 214974 "" "Prometheus/2.33.4" 2024-04-15T22:28:31.564 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:31 smithi176 ceph-mon[25776]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:31.564 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:31 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:31.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:31 smithi092 ceph-mon[24813]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:31.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:31 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:31.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:31 smithi092 ceph-mon[29727]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:31.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:31 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:32.627 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-04-15T22:28:32.627 INFO:teuthology.orchestra.run.smithi092.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", 2024-04-15T22:28:32.627 INFO:teuthology.orchestra.run.smithi092.stdout: "in_progress": true, 2024-04-15T22:28:32.628 INFO:teuthology.orchestra.run.smithi092.stdout: "services_complete": [], 2024-04-15T22:28:32.628 INFO:teuthology.orchestra.run.smithi092.stdout: "progress": "", 2024-04-15T22:28:32.628 INFO:teuthology.orchestra.run.smithi092.stdout: "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-15T22:28:32.628 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-04-15T22:28:33.409 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph health detail' 2024-04-15T22:28:33.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:33 smithi092 ceph-mon[29727]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:33.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:33 smithi092 ceph-mon[29727]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:28:33.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:33 smithi092 ceph-mon[24813]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:33.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:33 smithi092 ceph-mon[24813]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:28:33.714 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:28:33 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=error ts=2024-04-15T22:28:33.524Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.176:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.176 because it doesn't contain any IP SANs; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.92:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.92 because it doesn't contain any IP SANs" 2024-04-15T22:28:33.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:33 smithi176 ceph-mon[25776]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:33.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:33 smithi176 ceph-mon[25776]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T22:28:34.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:28:33 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:28:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:28:35.463 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:35 smithi092 ceph-mon[29727]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:35.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:35 smithi092 ceph-mon[24813]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:35.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:35 smithi176 ceph-mon[25776]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:37.274 INFO:teuthology.orchestra.run.smithi092.stdout:HEALTH_WARN Upgrade: failed to pull target image 2024-04-15T22:28:37.275 INFO:teuthology.orchestra.run.smithi092.stdout:[WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T22:28:37.275 INFO:teuthology.orchestra.run.smithi092.stdout: host smithi092 `cephadm pull` failed: cephadm exited with an error code: 1, stderr: Pulling container image quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765... 2024-04-15T22:28:37.275 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 125 from /bin/podman pull quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:28:37.275 INFO:teuthology.orchestra.run.smithi092.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765... 2024-04-15T22:28:37.275 INFO:teuthology.orchestra.run.smithi092.stdout:/bin/podman: stderr Error: initializing source docker://quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765: reading manifest a9a752df26c63acad72e1b3569fd79a515ca0765 in quay.ceph.io/ceph-ci/ceph: manifest unknown 2024-04-15T22:28:37.275 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: Failed command: /bin/podman pull quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T22:28:37.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:37 smithi092 ceph-mon[29727]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:37.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:37 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/3246325642' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-15T22:28:37.581 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:37 smithi092 ceph-mon[24813]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:37.582 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:37 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/3246325642' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-15T22:28:37.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:37 smithi176 ceph-mon[25776]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:37.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:37 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/3246325642' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-15T22:28:37.982 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-15T22:28:39.434 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:39 smithi092 ceph-mon[29727]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:39.434 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:39 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:39.434 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:39 smithi092 ceph-mon[29727]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:39.435 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:39 smithi092 ceph-mon[24813]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:39.435 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:39 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:39.435 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:39 smithi092 ceph-mon[24813]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:39.435 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:39 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 172.21.15.92 - - [15/Apr/2024:22:28:39] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-15T22:28:39.727 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:39 smithi176 ceph-mon[25776]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:39.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:39 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:39.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:39 smithi176 ceph-mon[25776]: from='mgr.24340 ' entity='mgr.y' 2024-04-15T22:28:40.639 INFO:teuthology.orchestra.run.smithi092.stdout:true 2024-04-15T22:28:40.639 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:40 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:28:40] "GET /metrics HTTP/1.1" 200 214965 "" "Prometheus/2.33.4" 2024-04-15T22:28:41.402 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-15T22:28:41.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:41 smithi092 ceph-mon[29727]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:41.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:41 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2537025842' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:41.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:41 smithi092 ceph-mon[29727]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:41.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:41 smithi092 ceph-mon[24813]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:41.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:41 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2537025842' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:41.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:41 smithi092 ceph-mon[24813]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:41.727 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:41 smithi176 ceph-mon[25776]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T22:28:41.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:41 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2537025842' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:41.728 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:41 smithi176 ceph-mon[25776]: from='client.24805 -' entity='client.iscsi.foo.smithi092.hsbujv' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T22:28:43.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:43 smithi092 ceph-mon[24813]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:43.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:43 smithi092 ceph-mon[29727]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:43.727 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:43 smithi176 ceph-mon[25776]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T22:28:44.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:28:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x[27220]: ::ffff:172.21.15.176 - - [15/Apr/2024:22:28:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T22:28:44.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:28:44.321 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi092 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-15T22:28:44.548 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4835b59349454ef2b34618d3e60df80c Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi092 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-15T22:28:44.550 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-15T22:28:44.560 INFO:tasks.cephadm:Teardown begin 2024-04-15T22:28:44.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:28:44.596 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:44 smithi092 ceph-mon[24813]: from='client.? 172.21.15.92:0/2397001024' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:44.597 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:44 smithi092 ceph-mon[29727]: from='client.? 172.21.15.92:0/2397001024' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:44.602 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:28:44.638 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:44 smithi176 ceph-mon[25776]: from='client.? 172.21.15.92:0/2397001024' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T22:28:44.643 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-15T22:28:44.643 DEBUG:teuthology.orchestra.run.smithi092:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-15T22:28:44.662 DEBUG:teuthology.orchestra.run.smithi176:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-15T22:28:44.701 INFO:tasks.cephadm:Stopping all daemons... 2024-04-15T22:28:44.702 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-15T22:28:44.702 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.a 2024-04-15T22:28:44.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:44 smithi092 systemd[1]: Stopping Ceph mon.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:44.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:44 smithi092 bash[62154]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon.a" found: no such container 2024-04-15T22:28:45.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:44 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a[24798]: 2024-04-15T22:28:44.901+0000 7f4e30b45700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:28:45.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:44 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a[24798]: 2024-04-15T22:28:44.901+0000 7f4e30b45700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-15T22:28:45.213 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:45 smithi092 podman[62166]: 2024-04-15 22:28:45.147782794 +0000 UTC m=+0.349063930 container died 5e439f1bb752bb2192198e1ffe51fa2382d8db60294210f0565d6c20771214da (image=quay.io/ceph/ceph:v17.2.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, distribution-scope=public, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, architecture=x86_64, version=8, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream) 2024-04-15T22:28:45.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:45 smithi092 podman[62166]: 2024-04-15 22:28:45.542440299 +0000 UTC m=+0.743721465 container cleanup 5e439f1bb752bb2192198e1ffe51fa2382d8db60294210f0565d6c20771214da (image=quay.io/ceph/ceph:v17.2.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, architecture=x86_64, io.buildah.version=1.19.8, name=centos-stream, distribution-scope=public, ceph=True, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-15T22:28:45.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:45 smithi092 bash[62166]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-a 2024-04-15T22:28:45.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Apr 15 22:28:45 smithi092 bash[62193]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon.a" found: no such container 2024-04-15T22:28:45.815 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.a.service' 2024-04-15T22:28:45.883 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:28:45.883 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-15T22:28:45.883 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-15T22:28:45.884 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.c 2024-04-15T22:28:46.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:45 smithi092 systemd[1]: Stopping Ceph mon.c for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:46.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:45 smithi092 bash[62245]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon.c" found: no such container 2024-04-15T22:28:46.455 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:46 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-c[29710]: 2024-04-15T22:28:46.104+0000 7fefd4404700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:28:46.455 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:46 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-c[29710]: 2024-04-15T22:28:46.104+0000 7fefd4404700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-15T22:28:46.455 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:46 smithi092 podman[62258]: 2024-04-15 22:28:46.156288295 +0000 UTC m=+0.186049888 container died 69aa2c1b085d0d7f19b354ec43764b665296c7a116e1f949ce606ee1531567b1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-c, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, version=8, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, GIT_CLEAN=True, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, ceph=True, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-15T22:28:46.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:46 smithi092 podman[62258]: 2024-04-15 22:28:46.5557865 +0000 UTC m=+0.585548085 container cleanup 69aa2c1b085d0d7f19b354ec43764b665296c7a116e1f949ce606ee1531567b1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-c, name=centos-stream, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, architecture=x86_64, release=754, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-15T22:28:46.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:46 smithi092 bash[62258]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-c 2024-04-15T22:28:46.713 INFO:journalctl@ceph.mon.c.smithi092.stdout:Apr 15 22:28:46 smithi092 bash[62285]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon.c" found: no such container 2024-04-15T22:28:46.818 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.c.service' 2024-04-15T22:28:46.857 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:28:46.858 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-15T22:28:46.858 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-15T22:28:46.858 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.b 2024-04-15T22:28:47.171 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:46 smithi176 systemd[1]: Stopping Ceph mon.b for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:47.172 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:46 smithi176 bash[51375]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon.b" found: no such container 2024-04-15T22:28:47.172 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:47 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-b[25750]: 2024-04-15T22:28:47.058+0000 7fec55459700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:28:47.172 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:47 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-b[25750]: 2024-04-15T22:28:47.058+0000 7fec55459700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-15T22:28:47.477 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:47 smithi176 podman[51388]: 2024-04-15 22:28:47.172593054 +0000 UTC m=+0.231202040 container died a93cba3b76ed50605e5c7cf16143143196d6b77475421631694edef158b9c0d1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-b, release=754, ceph=True, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-15T22:28:47.807 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.b.service' 2024-04-15T22:28:47.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:47 smithi176 podman[51388]: 2024-04-15 22:28:47.57988663 +0000 UTC m=+0.638495658 container cleanup a93cba3b76ed50605e5c7cf16143143196d6b77475421631694edef158b9c0d1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-b, GIT_CLEAN=True, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, distribution-scope=public, io.buildah.version=1.19.8, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, vendor=Red Hat, Inc., release=754, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870) 2024-04-15T22:28:47.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:47 smithi176 bash[51388]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-b 2024-04-15T22:28:47.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:47 smithi176 bash[51414]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon.b" found: no such container 2024-04-15T22:28:47.833 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:47 smithi176 podman[51402]: 2024-04-15 22:28:47.8032535 +0000 UTC m=+0.627485266 container remove a93cba3b76ed50605e5c7cf16143143196d6b77475421631694edef158b9c0d1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mon-b, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, version=8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., distribution-scope=public, io.openshift.expose-services=, ceph=True, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8) 2024-04-15T22:28:47.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:47 smithi176 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-15T22:28:47.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:47 smithi176 systemd[1]: Stopped Ceph mon.b for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:28:47.834 INFO:journalctl@ceph.mon.b.smithi176.stdout:Apr 15 22:28:47 smithi176 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mon.b.service: Consumed 7.094s CPU time. 2024-04-15T22:28:47.859 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:28:47.859 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-15T22:28:47.859 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-15T22:28:47.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mgr.y 2024-04-15T22:28:48.213 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:47 smithi092 systemd[1]: Stopping Ceph mgr.y for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:48.213 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:47 smithi092 bash[62340]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr.y" found: no such container 2024-04-15T22:28:48.213 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:48 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y[25044]: 2024-04-15T22:28:48.059+0000 7f6674d58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:28:48.213 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:48 smithi092 podman[62353]: 2024-04-15 22:28:48.090843966 +0000 UTC m=+0.147437098 container died 4e75752c42e97763cc95af53694c6755a03cac8e08c227584d334bb53cb37819 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, ceph=True, io.buildah.version=1.19.8, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, distribution-scope=public, name=centos-stream, build-date=2022-05-03T08:36:31.336870, release=754, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, architecture=x86_64) 2024-04-15T22:28:48.713 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:48 smithi092 podman[62353]: 2024-04-15 22:28:48.584705698 +0000 UTC m=+0.641298844 container cleanup 4e75752c42e97763cc95af53694c6755a03cac8e08c227584d334bb53cb37819 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y, vcs-type=git, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.openshift.expose-services=, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, ceph=True, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, version=8) 2024-04-15T22:28:48.713 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:48 smithi092 bash[62353]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-y 2024-04-15T22:28:48.713 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Apr 15 22:28:48 smithi092 bash[62381]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr.y" found: no such container 2024-04-15T22:28:48.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mgr.y.service' 2024-04-15T22:28:48.904 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:28:48.904 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-15T22:28:48.904 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-15T22:28:48.904 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mgr.x 2024-04-15T22:28:49.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:28:48 smithi176 systemd[1]: Stopping Ceph mgr.x for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:49.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:28:48 smithi176 bash[51468]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr.x" found: no such container 2024-04-15T22:28:49.228 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:28:49 smithi176 podman[51481]: 2024-04-15 22:28:49.145832185 +0000 UTC m=+0.152033662 container died 6460f0b4f8503913bd92a1889d68f8e8df2c96921801a9c98a851b4115142b50 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, release=754, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, architecture=x86_64, io.openshift.tags=base centos centos-stream, ceph=True, name=centos-stream, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, distribution-scope=public, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-15T22:28:49.681 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:28:49 smithi176 podman[51481]: 2024-04-15 22:28:49.447663676 +0000 UTC m=+0.453865155 container cleanup 6460f0b4f8503913bd92a1889d68f8e8df2c96921801a9c98a851b4115142b50 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, RELEASE=HEAD, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, com.redhat.component=centos-stream-container, release=754, build-date=2022-05-03T08:36:31.336870, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64) 2024-04-15T22:28:49.681 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:28:49 smithi176 bash[51481]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr-x 2024-04-15T22:28:49.681 INFO:journalctl@ceph.mgr.x.smithi176.stdout:Apr 15 22:28:49 smithi176 bash[51528]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-mgr.x" found: no such container 2024-04-15T22:28:49.686 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@mgr.x.service' 2024-04-15T22:28:49.727 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:28:49.728 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-15T22:28:49.728 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-15T22:28:49.728 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.0 2024-04-15T22:28:50.213 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:49 smithi092 systemd[1]: Stopping Ceph osd.0 for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:28:50.213 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:49 smithi092 bash[62435]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T22:28:50.213 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:49 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0[33002]: 2024-04-15T22:28:49.938+0000 7f4ebf7e5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:28:50.213 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:49 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0[33002]: 2024-04-15T22:28:49.938+0000 7f4ebf7e5700 -1 osd.0 96 *** Got signal Terminated *** 2024-04-15T22:28:50.214 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:49 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0[33002]: 2024-04-15T22:28:49.938+0000 7f4ebf7e5700 -1 osd.0 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:28:55.669 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:55 smithi092 podman[62448]: 2024-04-15 22:28:55.375283755 +0000 UTC m=+5.560435033 container died 77d7f9c5f13e4eb7b29b11ae39df9bdf3cb69c30b190520a9b9f8baa5168112d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., vcs-type=git, architecture=x86_64, release=754, name=centos-stream, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , version=8) 2024-04-15T22:28:55.963 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:55 smithi092 podman[62448]: 2024-04-15 22:28:55.870056941 +0000 UTC m=+6.055208265 container cleanup 77d7f9c5f13e4eb7b29b11ae39df9bdf3cb69c30b190520a9b9f8baa5168112d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.openshift.expose-services=, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, distribution-scope=public, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, architecture=x86_64, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, CEPH_POINT_RELEASE=-17.2.0) 2024-04-15T22:28:55.963 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:55 smithi092 bash[62448]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0 2024-04-15T22:28:56.404 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:56 smithi092 podman[62894]: 2024-04-15 22:28:56.071484606 +0000 UTC m=+0.695659432 container remove 77d7f9c5f13e4eb7b29b11ae39df9bdf3cb69c30b190520a9b9f8baa5168112d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, io.buildah.version=1.19.8, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vcs-type=git, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD) 2024-04-15T22:28:56.404 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:56 smithi092 bash[62937]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T22:28:57.963 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:57 smithi092 podman[63092]: 2024-04-15 22:28:57.653616315 +0000 UTC m=+0.019029675 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:28:58.367 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:58 smithi092 podman[63092]: 2024-04-15 22:28:58.107256724 +0000 UTC m=+0.472670131 container create fc03356ba7d2d6dfc3ff2db4aa455c2a16b864f93a49a696bb2af348ea3a6fa8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0-deactivate, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-type=git, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, architecture=x86_64, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, name=centos-stream, io.openshift.expose-services=, version=8, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, vendor=Red Hat, Inc.) 2024-04-15T22:28:58.653 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:58 smithi092 podman[63092]: 2024-04-15 22:28:58.368848075 +0000 UTC m=+0.734261433 container init fc03356ba7d2d6dfc3ff2db4aa455c2a16b864f93a49a696bb2af348ea3a6fa8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0-deactivate, architecture=x86_64, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , release=754, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, distribution-scope=public, ceph=True, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870) 2024-04-15T22:28:58.654 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:58 smithi092 podman[63092]: 2024-04-15 22:28:58.372486103 +0000 UTC m=+0.737899460 container start fc03356ba7d2d6dfc3ff2db4aa455c2a16b864f93a49a696bb2af348ea3a6fa8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0-deactivate, release=754, distribution-scope=public, vcs-type=git, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, name=centos-stream, architecture=x86_64, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True) 2024-04-15T22:28:58.654 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:58 smithi092 podman[63092]: 2024-04-15 22:28:58.46814192 +0000 UTC m=+0.833555279 container attach fc03356ba7d2d6dfc3ff2db4aa455c2a16b864f93a49a696bb2af348ea3a6fa8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.openshift.expose-services=, io.buildah.version=1.19.8, release=754, io.k8s.display-name=CentOS Stream 8, vcs-type=git, ceph=True, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., RELEASE=HEAD, distribution-scope=public, GIT_BRANCH=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_CLEAN=True) 2024-04-15T22:28:58.913 INFO:journalctl@ceph.osd.0.smithi092.stdout:Apr 15 22:28:58 smithi092 podman[63092]: 2024-04-15 22:28:58.795931819 +0000 UTC m=+1.161345160 container died fc03356ba7d2d6dfc3ff2db4aa455c2a16b864f93a49a696bb2af348ea3a6fa8 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-0-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , name=centos-stream, architecture=x86_64, version=8, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD) 2024-04-15T22:28:59.653 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.0.service' 2024-04-15T22:28:59.696 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:28:59.696 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-15T22:28:59.696 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-15T22:28:59.696 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.1 2024-04-15T22:28:59.893 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:28:59 smithi092 systemd[1]: Stopping Ceph osd.1 for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:29:00.213 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:28:59 smithi092 bash[63280]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T22:29:00.213 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1[35886]: 2024-04-15T22:29:00.027+0000 7f9fa4aea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:29:00.213 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1[35886]: 2024-04-15T22:29:00.027+0000 7f9fa4aea700 -1 osd.1 96 *** Got signal Terminated *** 2024-04-15T22:29:00.214 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:00 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1[35886]: 2024-04-15T22:29:00.027+0000 7f9fa4aea700 -1 osd.1 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:29:05.685 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:05 smithi092 podman[63294]: 2024-04-15 22:29:05.382383876 +0000 UTC m=+5.478780473 container died c1a9a397f575045aec86676a25cea04aca5bc7e0e7d86992588763d77b6a5408 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, io.openshift.expose-services=, GIT_CLEAN=True, release=754, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, name=centos-stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, distribution-scope=public, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-15T22:29:05.963 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:05 smithi092 podman[63294]: 2024-04-15 22:29:05.876253393 +0000 UTC m=+5.972650029 container cleanup c1a9a397f575045aec86676a25cea04aca5bc7e0e7d86992588763d77b6a5408 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, build-date=2022-05-03T08:36:31.336870, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, ceph=True, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, RELEASE=HEAD) 2024-04-15T22:29:05.963 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:05 smithi092 bash[63294]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1 2024-04-15T22:29:05.963 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:05 smithi092 bash[63420]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T22:29:06.393 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:06 smithi092 podman[63406]: 2024-04-15 22:29:06.144179927 +0000 UTC m=+0.760472209 container remove c1a9a397f575045aec86676a25cea04aca5bc7e0e7d86992588763d77b6a5408 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, distribution-scope=public, architecture=x86_64, io.openshift.tags=base centos centos-stream, release=754, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8) 2024-04-15T22:29:06.394 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:06 smithi092 podman[63480]: 2024-04-15 22:29:06.274924171 +0000 UTC m=+0.019387442 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:29:06.394 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:06 smithi092 podman[63480]: 2024-04-15 22:29:06.395087589 +0000 UTC m=+0.139550889 container create d88f1ab9ba0daf9aee68c2aaf48236c596db12c638428d24ff7a0751e16e4034 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1-deactivate, version=8, io.openshift.tags=base centos centos-stream, distribution-scope=public, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, name=centos-stream, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.buildah.version=1.19.8, ceph=True, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, architecture=x86_64) 2024-04-15T22:29:06.690 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:06 smithi092 podman[63480]: 2024-04-15 22:29:06.588753385 +0000 UTC m=+0.333216643 container init d88f1ab9ba0daf9aee68c2aaf48236c596db12c638428d24ff7a0751e16e4034 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, name=centos-stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, ceph=True) 2024-04-15T22:29:06.690 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:06 smithi092 podman[63480]: 2024-04-15 22:29:06.592055941 +0000 UTC m=+0.336519200 container start d88f1ab9ba0daf9aee68c2aaf48236c596db12c638428d24ff7a0751e16e4034 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1-deactivate, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, version=8, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public) 2024-04-15T22:29:06.690 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:06 smithi092 podman[63480]: 2024-04-15 22:29:06.688771512 +0000 UTC m=+0.433234786 container attach d88f1ab9ba0daf9aee68c2aaf48236c596db12c638428d24ff7a0751e16e4034 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1-deactivate, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, RELEASE=HEAD, io.openshift.expose-services=, version=8, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-15T22:29:06.963 INFO:journalctl@ceph.osd.1.smithi092.stdout:Apr 15 22:29:06 smithi092 podman[63480]: 2024-04-15 22:29:06.880458217 +0000 UTC m=+0.624921469 container died d88f1ab9ba0daf9aee68c2aaf48236c596db12c638428d24ff7a0751e16e4034 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-1-deactivate, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, version=8, com.redhat.component=centos-stream-container, distribution-scope=public, vcs-type=git, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, RELEASE=HEAD, GIT_CLEAN=True, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.buildah.version=1.19.8, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-15T22:29:07.602 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.1.service' 2024-04-15T22:29:07.647 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:29:07.647 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-15T22:29:07.647 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-15T22:29:07.648 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.2 2024-04-15T22:29:07.861 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:07 smithi092 systemd[1]: Stopping Ceph osd.2 for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:29:07.861 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:07 smithi092 bash[63558]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T22:29:08.213 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:07 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2[38836]: 2024-04-15T22:29:07.861+0000 7f8cee9d8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:29:08.213 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:07 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2[38836]: 2024-04-15T22:29:07.861+0000 7f8cee9d8700 -1 osd.2 96 *** Got signal Terminated *** 2024-04-15T22:29:08.213 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:07 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2[38836]: 2024-04-15T22:29:07.861+0000 7f8cee9d8700 -1 osd.2 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:29:13.662 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:13 smithi092 podman[63571]: 2024-04-15 22:29:13.397566298 +0000 UTC m=+5.636262395 container died 7e162849c1f7037cafb9acc68c9e27cb30111b391df88ba89fdbe6132e1bb8c3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, architecture=x86_64, vcs-type=git, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, release=754, RELEASE=HEAD, io.buildah.version=1.19.8, version=8, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD) 2024-04-15T22:29:13.963 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:13 smithi092 podman[63571]: 2024-04-15 22:29:13.828108403 +0000 UTC m=+6.066804549 container cleanup 7e162849c1f7037cafb9acc68c9e27cb30111b391df88ba89fdbe6132e1bb8c3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, distribution-scope=public, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, architecture=x86_64, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=) 2024-04-15T22:29:13.963 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:13 smithi092 bash[63571]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2 2024-04-15T22:29:14.269 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:14 smithi092 podman[63590]: 2024-04-15 22:29:14.020972189 +0000 UTC m=+0.617719451 container remove 7e162849c1f7037cafb9acc68c9e27cb30111b391df88ba89fdbe6132e1bb8c3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.openshift.expose-services=, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, release=754, version=8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8) 2024-04-15T22:29:14.269 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:14 smithi092 bash[63604]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T22:29:14.556 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:14 smithi092 podman[63668]: 2024-04-15 22:29:14.174642638 +0000 UTC m=+0.018660683 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:29:14.556 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:14 smithi092 podman[63668]: 2024-04-15 22:29:14.296714616 +0000 UTC m=+0.140732638 container create ddc3db323689b5783730d871dd742257f3be7fbbcfa877b076ada81b063c722d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2-deactivate, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, release=754, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., ceph=True, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870) 2024-04-15T22:29:14.556 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:14 smithi092 podman[63668]: 2024-04-15 22:29:14.482496782 +0000 UTC m=+0.326514802 container init ddc3db323689b5783730d871dd742257f3be7fbbcfa877b076ada81b063c722d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2-deactivate, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, RELEASE=HEAD, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8) 2024-04-15T22:29:14.556 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:14 smithi092 podman[63668]: 2024-04-15 22:29:14.486173917 +0000 UTC m=+0.330191939 container start ddc3db323689b5783730d871dd742257f3be7fbbcfa877b076ada81b063c722d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, distribution-scope=public, ceph=True, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, architecture=x86_64, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0) 2024-04-15T22:29:14.557 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:14 smithi092 podman[63668]: 2024-04-15 22:29:14.557324579 +0000 UTC m=+0.401342604 container attach ddc3db323689b5783730d871dd742257f3be7fbbcfa877b076ada81b063c722d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.openshift.expose-services=, version=8, architecture=x86_64, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, vendor=Red Hat, Inc.) 2024-04-15T22:29:14.963 INFO:journalctl@ceph.osd.2.smithi092.stdout:Apr 15 22:29:14 smithi092 podman[63668]: 2024-04-15 22:29:14.775025684 +0000 UTC m=+0.619043713 container died ddc3db323689b5783730d871dd742257f3be7fbbcfa877b076ada81b063c722d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-2-deactivate, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.openshift.expose-services=, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, distribution-scope=public) 2024-04-15T22:29:15.332 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.2.service' 2024-04-15T22:29:15.374 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:29:15.374 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-15T22:29:15.375 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-15T22:29:15.375 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.3 2024-04-15T22:29:15.595 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:15 smithi092 systemd[1]: Stopping Ceph osd.3 for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:29:15.596 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:15 smithi092 bash[63745]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T22:29:15.963 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:15 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3[41854]: 2024-04-15T22:29:15.595+0000 7f7fe9298700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:29:15.963 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:15 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3[41854]: 2024-04-15T22:29:15.595+0000 7f7fe9298700 -1 osd.3 96 *** Got signal Terminated *** 2024-04-15T22:29:15.963 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:15 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3[41854]: 2024-04-15T22:29:15.595+0000 7f7fe9298700 -1 osd.3 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:29:16.663 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:16 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:16.469+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:50.770442+0000 front 2024-04-15T22:28:50.770437+0000 (oldest deadline 2024-04-15T22:29:15.470460+0000) 2024-04-15T22:29:16.663 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:16 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:16.344+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:16.977 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:16 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:16.662+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:17.686 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:17 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:17.445+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:50.770442+0000 front 2024-04-15T22:28:50.770437+0000 (oldest deadline 2024-04-15T22:29:15.470460+0000) 2024-04-15T22:29:17.687 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:17 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:17.297+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:17.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:17 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:17.686+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:18.679 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:18 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:18.434+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:50.770442+0000 front 2024-04-15T22:28:50.770437+0000 (oldest deadline 2024-04-15T22:29:15.470460+0000) 2024-04-15T22:29:18.679 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:18 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:18.297+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:18.977 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:18 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:18.679+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:18.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:18 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:18.805+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:19.707 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:19 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:19.392+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:50.770442+0000 front 2024-04-15T22:28:50.770437+0000 (oldest deadline 2024-04-15T22:29:15.470460+0000) 2024-04-15T22:29:19.707 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:19 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:19.315+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:19.977 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:19 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:19.708+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:19.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:19 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:19.759+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:20.463 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:20 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3[41854]: 2024-04-15T22:29:20.257+0000 7f7fe242c700 -1 osd.3 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:54.414355+0000 front 2024-04-15T22:28:54.414363+0000 (oldest deadline 2024-04-15T22:29:19.714551+0000) 2024-04-15T22:29:20.717 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:20.384+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:50.770442+0000 front 2024-04-15T22:28:50.770437+0000 (oldest deadline 2024-04-15T22:29:15.470460+0000) 2024-04-15T22:29:20.718 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:20.279+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:20.977 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:20.717+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:20.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:20 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:20.805+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:21.182 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:20 smithi092 podman[63758]: 2024-04-15 22:29:20.976931098 +0000 UTC m=+5.484168372 container died c26f54e54812eb406910f662499a0a209bd7a77edeeaf42d49d3745bf0f4eb37 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3, name=centos-stream, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, architecture=x86_64, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, com.redhat.component=centos-stream-container, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, version=8) 2024-04-15T22:29:21.463 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:21 smithi092 podman[63758]: 2024-04-15 22:29:21.358492929 +0000 UTC m=+5.865730210 container cleanup c26f54e54812eb406910f662499a0a209bd7a77edeeaf42d49d3745bf0f4eb37 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, version=8, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream) 2024-04-15T22:29:21.463 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:21 smithi092 bash[63758]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3 2024-04-15T22:29:21.463 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:21 smithi092 bash[64143]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T22:29:21.679 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:21.346+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:50.770442+0000 front 2024-04-15T22:28:50.770437+0000 (oldest deadline 2024-04-15T22:29:15.470460+0000) 2024-04-15T22:29:21.679 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:21.292+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:21.955 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:21 smithi092 podman[64130]: 2024-04-15 22:29:21.634929397 +0000 UTC m=+0.652742371 container remove c26f54e54812eb406910f662499a0a209bd7a77edeeaf42d49d3745bf0f4eb37 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, maintainer=Guillaume Abrioux , ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, release=754, io.openshift.tags=base centos centos-stream, vcs-type=git, io.buildah.version=1.19.8, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, distribution-scope=public) 2024-04-15T22:29:21.956 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:21 smithi092 podman[64229]: 2024-04-15 22:29:21.843834353 +0000 UTC m=+0.109479762 container create 203f1280cedb149f85cd507036ae006279f2a6c66f76ae158815bca2e6b9b3ce (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vcs-type=git, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, architecture=x86_64, com.redhat.component=centos-stream-container) 2024-04-15T22:29:21.956 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:21 smithi092 podman[64229]: 2024-04-15 22:29:21.759827265 +0000 UTC m=+0.025472690 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:29:21.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:21.679+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:21.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:21 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:21.798+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:22.213 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:22 smithi092 podman[64229]: 2024-04-15 22:29:22.08766141 +0000 UTC m=+0.353306823 container init 203f1280cedb149f85cd507036ae006279f2a6c66f76ae158815bca2e6b9b3ce (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3-deactivate, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.component=centos-stream-container, ceph=True) 2024-04-15T22:29:22.213 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:22 smithi092 podman[64229]: 2024-04-15 22:29:22.091421316 +0000 UTC m=+0.357066726 container start 203f1280cedb149f85cd507036ae006279f2a6c66f76ae158815bca2e6b9b3ce (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3-deactivate, io.openshift.expose-services=, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, version=8, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, RELEASE=HEAD, distribution-scope=public, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-15T22:29:22.213 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:22 smithi092 podman[64229]: 2024-04-15 22:29:22.171072798 +0000 UTC m=+0.436718222 container attach 203f1280cedb149f85cd507036ae006279f2a6c66f76ae158815bca2e6b9b3ce (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3-deactivate, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, ceph=True, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, release=754, architecture=x86_64, com.redhat.component=centos-stream-container, vcs-type=git, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0) 2024-04-15T22:29:22.633 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:22 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:22.380+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:50.770442+0000 front 2024-04-15T22:28:50.770437+0000 (oldest deadline 2024-04-15T22:29:15.470460+0000) 2024-04-15T22:29:22.634 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:22 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:22.313+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:22.689 INFO:journalctl@ceph.osd.3.smithi092.stdout:Apr 15 22:29:22 smithi092 podman[64229]: 2024-04-15 22:29:22.409327756 +0000 UTC m=+0.674973174 container died 203f1280cedb149f85cd507036ae006279f2a6c66f76ae158815bca2e6b9b3ce (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-3-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, GIT_BRANCH=HEAD, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, version=8, vendor=Red Hat, Inc., vcs-type=git) 2024-04-15T22:29:22.977 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:22 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:22.633+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:22.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:22 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:22.789+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:23.010 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.3.service' 2024-04-15T22:29:23.052 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:29:23.052 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-15T22:29:23.053 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-15T22:29:23.053 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.4 2024-04-15T22:29:23.339 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:23 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:23.285+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:23.340 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:23 smithi176 systemd[1]: Stopping Ceph osd.4 for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:29:23.340 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:23 smithi176 bash[52358]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T22:29:23.340 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:23 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:23.259+0000 7f37cb34b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:29:23.340 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:23 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:23.259+0000 7f37cb34b700 -1 osd.4 96 *** Got signal Terminated *** 2024-04-15T22:29:23.340 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:23 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:23.259+0000 7f37cb34b700 -1 osd.4 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:29:23.340 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:23 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:23.340+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:50.770442+0000 front 2024-04-15T22:28:50.770437+0000 (oldest deadline 2024-04-15T22:29:15.470460+0000) 2024-04-15T22:29:23.977 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:23 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:23.620+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:23.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:23 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:23.817+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:24.659 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:24 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:24.382+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:50.770442+0000 front 2024-04-15T22:28:50.770437+0000 (oldest deadline 2024-04-15T22:29:15.470460+0000) 2024-04-15T22:29:24.659 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:24 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:24.283+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:24.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:24 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:24.660+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:24.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:24 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:24.851+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:25.616 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:25.368+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:50.770442+0000 front 2024-04-15T22:28:50.770437+0000 (oldest deadline 2024-04-15T22:29:15.470460+0000) 2024-04-15T22:29:25.616 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:25.327+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:25.867 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:25.617+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:26.227 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:25 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:25.868+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:26.727 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:26.589+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:26.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:26.360+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:26.728 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:26.391+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:50.770442+0000 front 2024-04-15T22:28:50.770437+0000 (oldest deadline 2024-04-15T22:29:15.470460+0000) 2024-04-15T22:29:26.728 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:26.391+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:28:59.571585+0000 front 2024-04-15T22:28:59.571562+0000 (oldest deadline 2024-04-15T22:29:25.471523+0000) 2024-04-15T22:29:27.227 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:26 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:26.853+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:27.630 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:27 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:27.314+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:27.631 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:27 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:27.349+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:50.770442+0000 front 2024-04-15T22:28:50.770437+0000 (oldest deadline 2024-04-15T22:29:15.470460+0000) 2024-04-15T22:29:27.631 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:27 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4[30413]: 2024-04-15T22:29:27.349+0000 7f37c44df700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:28:59.571585+0000 front 2024-04-15T22:28:59.571562+0000 (oldest deadline 2024-04-15T22:29:25.471523+0000) 2024-04-15T22:29:27.977 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:27 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:27.631+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:27.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:27 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:27.836+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:28.582 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:28 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:28.280+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:28.583 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:28 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:28.280+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:28.968 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:28 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:28.828+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:28.969 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:28 smithi176 podman[52369]: 2024-04-15 22:29:28.679257661 +0000 UTC m=+5.533676646 container died 300e4da7c469fba25f32d250c510e56e0b389f9c7ef14bf650ea706ff1bdc4aa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., architecture=x86_64, RELEASE=HEAD, ceph=True, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, name=centos-stream, version=8, io.openshift.expose-services=, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-15T22:29:28.969 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:28 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:28.583+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:28.969 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:28 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:28.583+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:03.672955+0000 front 2024-04-15T22:29:03.673031+0000 (oldest deadline 2024-04-15T22:29:27.772808+0000) 2024-04-15T22:29:29.228 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:29 smithi176 podman[52369]: 2024-04-15 22:29:29.118219662 +0000 UTC m=+5.972638686 container cleanup 300e4da7c469fba25f32d250c510e56e0b389f9c7ef14bf650ea706ff1bdc4aa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , ceph=True, architecture=x86_64, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vcs-type=git, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, vendor=Red Hat, Inc.) 2024-04-15T22:29:29.228 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:29 smithi176 bash[52369]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4 2024-04-15T22:29:29.593 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:29 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:29.314+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:29.593 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:29 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:29.314+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:29.593 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:29 smithi176 podman[52387]: 2024-04-15 22:29:29.327959837 +0000 UTC m=+0.648640349 container remove 300e4da7c469fba25f32d250c510e56e0b389f9c7ef14bf650ea706ff1bdc4aa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4, architecture=x86_64, com.redhat.component=centos-stream-container, version=8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, GIT_BRANCH=HEAD, distribution-scope=public, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-15T22:29:29.593 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:29 smithi176 bash[52413]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T22:29:29.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:29 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:29.788+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:29.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:29 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:29.618+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:29.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:29 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:29.618+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:03.672955+0000 front 2024-04-15T22:29:03.673031+0000 (oldest deadline 2024-04-15T22:29:27.772808+0000) 2024-04-15T22:29:30.441 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:30 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:30.314+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:30.441 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:30 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:30.314+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:30.727 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:30 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:30.613+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:30.728 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:30 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:30.613+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:03.672955+0000 front 2024-04-15T22:29:03.673031+0000 (oldest deadline 2024-04-15T22:29:27.772808+0000) 2024-04-15T22:29:30.728 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:30 smithi176 podman[52500]: 2024-04-15 22:29:30.509295818 +0000 UTC m=+0.019849727 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:29:30.728 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:30 smithi176 podman[52500]: 2024-04-15 22:29:30.635278142 +0000 UTC m=+0.145832026 container create fac7f1650a2787ecd50271d1a5572f273987f8c2c6f4f0dbcab0087b1251211b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4-deactivate, vcs-type=git, ceph=True, com.redhat.component=centos-stream-container, GIT_CLEAN=True, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, RELEASE=HEAD, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream) 2024-04-15T22:29:31.149 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:30 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:30.805+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:31.150 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:30 smithi176 podman[52500]: 2024-04-15 22:29:30.862850247 +0000 UTC m=+0.373404130 container init fac7f1650a2787ecd50271d1a5572f273987f8c2c6f4f0dbcab0087b1251211b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, distribution-scope=public, ceph=True, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.openshift.expose-services=, name=centos-stream, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8) 2024-04-15T22:29:31.150 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:30 smithi176 podman[52500]: 2024-04-15 22:29:30.866286569 +0000 UTC m=+0.376840452 container start fac7f1650a2787ecd50271d1a5572f273987f8c2c6f4f0dbcab0087b1251211b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=-17.2.0, release=754, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, version=8, name=centos-stream, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.openshift.expose-services=, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8) 2024-04-15T22:29:31.150 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:30 smithi176 podman[52500]: 2024-04-15 22:29:30.94638841 +0000 UTC m=+0.456942293 container attach fac7f1650a2787ecd50271d1a5572f273987f8c2c6f4f0dbcab0087b1251211b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4-deactivate, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, ceph=True, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, version=8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, release=754, GIT_BRANCH=HEAD, architecture=x86_64, distribution-scope=public) 2024-04-15T22:29:31.150 INFO:journalctl@ceph.osd.4.smithi176.stdout:Apr 15 22:29:31 smithi176 podman[52500]: 2024-04-15 22:29:31.151094682 +0000 UTC m=+0.661648585 container died fac7f1650a2787ecd50271d1a5572f273987f8c2c6f4f0dbcab0087b1251211b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, GIT_CLEAN=True, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , RELEASE=HEAD, version=8, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., ceph=True) 2024-04-15T22:29:31.423 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:31 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:31.271+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:31.423 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:31 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:31.271+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:31.699 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:31 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:31.631+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:31.699 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:31 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:31.631+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:03.672955+0000 front 2024-04-15T22:29:03.673031+0000 (oldest deadline 2024-04-15T22:29:27.772808+0000) 2024-04-15T22:29:31.716 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.4.service' 2024-04-15T22:29:31.760 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:29:31.760 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-15T22:29:31.760 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-15T22:29:31.760 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.5 2024-04-15T22:29:31.977 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:31 smithi176 systemd[1]: Stopping Ceph osd.5 for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:29:31.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:31 smithi176 bash[52589]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T22:29:31.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:31 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:31.978+0000 7fdab4574700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:29:31.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:31 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:31.813+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:31.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:31 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:31.813+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:04.648896+0000 front 2024-04-15T22:29:04.648904+0000 (oldest deadline 2024-04-15T22:29:31.049473+0000) 2024-04-15T22:29:32.227 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:31 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:31.978+0000 7fdab4574700 -1 osd.5 96 *** Got signal Terminated *** 2024-04-15T22:29:32.228 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:31 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:31.978+0000 7fdab4574700 -1 osd.5 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:29:32.638 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:32 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:32.282+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:32.638 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:32 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:32.282+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:32.977 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:32 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:32.638+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:32.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:32 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:32.638+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:03.672955+0000 front 2024-04-15T22:29:03.673031+0000 (oldest deadline 2024-04-15T22:29:27.772808+0000) 2024-04-15T22:29:32.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:32 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:32.826+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:32.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:32 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:32.826+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:04.648896+0000 front 2024-04-15T22:29:04.648904+0000 (oldest deadline 2024-04-15T22:29:31.049473+0000) 2024-04-15T22:29:33.638 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:33 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:33.332+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:33.638 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:33 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:33.332+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:33.977 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:33 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:33.662+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:33.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:33 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:33.662+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:03.672955+0000 front 2024-04-15T22:29:03.673031+0000 (oldest deadline 2024-04-15T22:29:27.772808+0000) 2024-04-15T22:29:33.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:33 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:33.817+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:33.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:33 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:33.817+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:04.648896+0000 front 2024-04-15T22:29:04.648904+0000 (oldest deadline 2024-04-15T22:29:31.049473+0000) 2024-04-15T22:29:34.709 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:34 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:34.287+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:34.710 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:34 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:34.287+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:34.977 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:34 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:34.710+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:34.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:34 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:34.710+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:03.672955+0000 front 2024-04-15T22:29:03.673031+0000 (oldest deadline 2024-04-15T22:29:27.772808+0000) 2024-04-15T22:29:34.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:34 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:34.839+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:34.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:34 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:34.839+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:04.648896+0000 front 2024-04-15T22:29:04.648904+0000 (oldest deadline 2024-04-15T22:29:31.049473+0000) 2024-04-15T22:29:34.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:34 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:34.839+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:11.049945+0000 front 2024-04-15T22:29:11.049907+0000 (oldest deadline 2024-04-15T22:29:33.850524+0000) 2024-04-15T22:29:35.477 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:35 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:35.266+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:35.478 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:35 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:35.266+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:35.932 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:35 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:35.715+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:35.932 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:35 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:35.715+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:03.672955+0000 front 2024-04-15T22:29:03.673031+0000 (oldest deadline 2024-04-15T22:29:27.772808+0000) 2024-04-15T22:29:35.932 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:35 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:35.854+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:35.932 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:35 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:35.854+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:04.648896+0000 front 2024-04-15T22:29:04.648904+0000 (oldest deadline 2024-04-15T22:29:31.049473+0000) 2024-04-15T22:29:35.932 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:35 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:35.854+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:11.049945+0000 front 2024-04-15T22:29:11.049907+0000 (oldest deadline 2024-04-15T22:29:33.850524+0000) 2024-04-15T22:29:36.726 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:36.246+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:36.726 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:36.246+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:36.726 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:36.246+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:36.977 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:36.892+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:36.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:36.892+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:04.648896+0000 front 2024-04-15T22:29:04.648904+0000 (oldest deadline 2024-04-15T22:29:31.049473+0000) 2024-04-15T22:29:36.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:36.892+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:11.049945+0000 front 2024-04-15T22:29:11.049907+0000 (oldest deadline 2024-04-15T22:29:33.850524+0000) 2024-04-15T22:29:36.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:36.727+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.670778+0000 front 2024-04-15T22:28:52.670784+0000 (oldest deadline 2024-04-15T22:29:16.170790+0000) 2024-04-15T22:29:36.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:36.727+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:03.672955+0000 front 2024-04-15T22:29:03.673031+0000 (oldest deadline 2024-04-15T22:29:27.772808+0000) 2024-04-15T22:29:36.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:36 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5[33314]: 2024-04-15T22:29:36.727+0000 7fdaaf70c700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:12.473888+0000 front 2024-04-15T22:29:12.473933+0000 (oldest deadline 2024-04-15T22:29:36.474259+0000) 2024-04-15T22:29:37.727 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:37 smithi176 podman[52602]: 2024-04-15 22:29:37.458652473 +0000 UTC m=+5.580671214 container died 9ba0015eb74ac94625332c04d286ec15f45aaac8c5a5d639d3b32c47a71c114a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, ceph=True, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, distribution-scope=public, vendor=Red Hat, Inc.) 2024-04-15T22:29:37.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:37.230+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:37.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:37.230+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:37.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:37.230+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:37.990 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:37.864+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:37.990 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:37.864+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:04.648896+0000 front 2024-04-15T22:29:04.648904+0000 (oldest deadline 2024-04-15T22:29:31.049473+0000) 2024-04-15T22:29:37.990 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:37.865+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:11.049945+0000 front 2024-04-15T22:29:11.049907+0000 (oldest deadline 2024-04-15T22:29:33.850524+0000) 2024-04-15T22:29:37.990 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:37 smithi176 podman[52602]: 2024-04-15 22:29:37.945084332 +0000 UTC m=+6.067103102 container cleanup 9ba0015eb74ac94625332c04d286ec15f45aaac8c5a5d639d3b32c47a71c114a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., vcs-type=git, RELEASE=HEAD, distribution-scope=public, io.openshift.expose-services=, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, ceph=True, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-15T22:29:37.991 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:37 smithi176 bash[52602]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5 2024-04-15T22:29:37.991 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:37 smithi176 bash[53273]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T22:29:38.454 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:38 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:38.227+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:38.454 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:38 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:38.227+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:38.454 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:38 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:38.227+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:38.455 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:38 smithi176 podman[53260]: 2024-04-15 22:29:38.19630576 +0000 UTC m=+0.734845156 container remove 9ba0015eb74ac94625332c04d286ec15f45aaac8c5a5d639d3b32c47a71c114a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5, io.openshift.expose-services=, vcs-type=git, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, version=8, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, maintainer=Guillaume Abrioux ) 2024-04-15T22:29:38.455 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:38 smithi176 podman[53341]: 2024-04-15 22:29:38.344143804 +0000 UTC m=+0.019007460 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:29:38.455 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:38 smithi176 podman[53341]: 2024-04-15 22:29:38.455679763 +0000 UTC m=+0.130543392 container create a7cf0d7543f97bc29baeaec85b9c946767beb553d53011081dec4a38695f6364 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, release=754, GIT_BRANCH=HEAD, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, ceph=True, RELEASE=HEAD, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-15T22:29:38.715 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:38 smithi176 podman[53341]: 2024-04-15 22:29:38.641221088 +0000 UTC m=+0.316084717 container init a7cf0d7543f97bc29baeaec85b9c946767beb553d53011081dec4a38695f6364 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5-deactivate, vcs-type=git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., release=754, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, version=8, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-15T22:29:38.715 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:38 smithi176 podman[53341]: 2024-04-15 22:29:38.644709424 +0000 UTC m=+0.319573060 container start a7cf0d7543f97bc29baeaec85b9c946767beb553d53011081dec4a38695f6364 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5-deactivate, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, ceph=True, io.openshift.expose-services=, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, version=8, vendor=Red Hat, Inc., RELEASE=HEAD, io.openshift.tags=base centos centos-stream, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux ) 2024-04-15T22:29:38.715 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:38 smithi176 podman[53341]: 2024-04-15 22:29:38.716180869 +0000 UTC m=+0.391044517 container attach a7cf0d7543f97bc29baeaec85b9c946767beb553d53011081dec4a38695f6364 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vendor=Red Hat, Inc., release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, io.openshift.expose-services=, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0) 2024-04-15T22:29:38.977 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:38 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:38.817+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:38.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:38 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:38.817+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:04.648896+0000 front 2024-04-15T22:29:04.648904+0000 (oldest deadline 2024-04-15T22:29:31.049473+0000) 2024-04-15T22:29:38.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:38 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:38.817+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:11.049945+0000 front 2024-04-15T22:29:11.049907+0000 (oldest deadline 2024-04-15T22:29:33.850524+0000) 2024-04-15T22:29:38.978 INFO:journalctl@ceph.osd.5.smithi176.stdout:Apr 15 22:29:38 smithi176 podman[53341]: 2024-04-15 22:29:38.926007363 +0000 UTC m=+0.600871003 container died a7cf0d7543f97bc29baeaec85b9c946767beb553d53011081dec4a38695f6364 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-5-deactivate, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, name=centos-stream, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, vcs-type=git, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, vendor=Red Hat, Inc., distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0) 2024-04-15T22:29:39.477 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:39 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:39.273+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:39.478 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:39 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:39.273+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:39.478 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:39 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:39.273+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:39.574 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.5.service' 2024-04-15T22:29:39.623 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:29:39.624 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-15T22:29:39.624 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-15T22:29:39.624 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.6 2024-04-15T22:29:39.830 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:39 smithi176 systemd[1]: Stopping Ceph osd.6 for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:29:39.830 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:39 smithi176 bash[53421]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T22:29:40.228 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:39 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:39.831+0000 7f1e2f2ab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:29:40.228 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:39 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:39.831+0000 7f1e2f2ab700 -1 osd.6 96 *** Got signal Terminated *** 2024-04-15T22:29:40.228 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:39 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:39.831+0000 7f1e2f2ab700 -1 osd.6 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:29:40.228 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:39 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:39.857+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:40.228 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:39 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:39.857+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:04.648896+0000 front 2024-04-15T22:29:04.648904+0000 (oldest deadline 2024-04-15T22:29:31.049473+0000) 2024-04-15T22:29:40.228 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:39 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:39.857+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:11.049945+0000 front 2024-04-15T22:29:11.049907+0000 (oldest deadline 2024-04-15T22:29:33.850524+0000) 2024-04-15T22:29:40.663 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:40 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:40.246+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:40.663 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:40 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:40.246+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:40.664 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:40 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:40.246+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:40.977 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:40 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:40.876+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:40.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:40 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:40.876+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:04.648896+0000 front 2024-04-15T22:29:04.648904+0000 (oldest deadline 2024-04-15T22:29:31.049473+0000) 2024-04-15T22:29:40.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:40 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:40.876+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:11.049945+0000 front 2024-04-15T22:29:11.049907+0000 (oldest deadline 2024-04-15T22:29:33.850524+0000) 2024-04-15T22:29:41.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:41.284+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:41.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:41.284+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:41.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:41.284+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:41.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:41.284+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-04-15T22:29:15.690645+0000 front 2024-04-15T22:29:15.690649+0000 (oldest deadline 2024-04-15T22:29:40.990639+0000) 2024-04-15T22:29:42.227 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:41.909+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:42.228 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:41.909+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:04.648896+0000 front 2024-04-15T22:29:04.648904+0000 (oldest deadline 2024-04-15T22:29:31.049473+0000) 2024-04-15T22:29:42.228 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:41 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:41.909+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:11.049945+0000 front 2024-04-15T22:29:11.049907+0000 (oldest deadline 2024-04-15T22:29:33.850524+0000) 2024-04-15T22:29:42.727 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:42 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:42.243+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:42.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:42 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:42.243+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:42.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:42 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:42.243+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:42.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:42 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:42.243+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-04-15T22:29:15.690645+0000 front 2024-04-15T22:29:15.690649+0000 (oldest deadline 2024-04-15T22:29:40.990639+0000) 2024-04-15T22:29:43.227 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:42 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:42.884+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:43.228 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:42 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:42.884+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:04.648896+0000 front 2024-04-15T22:29:04.648904+0000 (oldest deadline 2024-04-15T22:29:31.049473+0000) 2024-04-15T22:29:43.228 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:42 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:42.884+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:11.049945+0000 front 2024-04-15T22:29:11.049907+0000 (oldest deadline 2024-04-15T22:29:33.850524+0000) 2024-04-15T22:29:43.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:43.292+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:43.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:43.292+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:43.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:43.292+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:43.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:43.292+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-04-15T22:29:15.690645+0000 front 2024-04-15T22:29:15.690649+0000 (oldest deadline 2024-04-15T22:29:40.990639+0000) 2024-04-15T22:29:44.227 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:43.853+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:53.547252+0000 front 2024-04-15T22:28:53.547195+0000 (oldest deadline 2024-04-15T22:29:18.247236+0000) 2024-04-15T22:29:44.228 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:43.853+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:04.648896+0000 front 2024-04-15T22:29:04.648904+0000 (oldest deadline 2024-04-15T22:29:31.049473+0000) 2024-04-15T22:29:44.228 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:43.853+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:11.049945+0000 front 2024-04-15T22:29:11.049907+0000 (oldest deadline 2024-04-15T22:29:33.850524+0000) 2024-04-15T22:29:44.228 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:43 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6[36229]: 2024-04-15T22:29:43.853+0000 7f1e2c447700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-04-15T22:29:19.652149+0000 front 2024-04-15T22:29:19.652065+0000 (oldest deadline 2024-04-15T22:29:43.751963+0000) 2024-04-15T22:29:44.727 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:44 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:44.272+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:44.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:44 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:44.272+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:44.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:44 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:44.272+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:44.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:44 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:44.272+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-04-15T22:29:15.690645+0000 front 2024-04-15T22:29:15.690649+0000 (oldest deadline 2024-04-15T22:29:40.990639+0000) 2024-04-15T22:29:45.638 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:45 smithi176 podman[53434]: 2024-04-15 22:29:45.343096717 +0000 UTC m=+5.631142947 container died 1727b9b7563530ba6cd7338f302d828e2da841eab441f1357c8a75c7ce8badfa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, name=centos-stream, ceph=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, vcs-type=git, RELEASE=HEAD) 2024-04-15T22:29:45.640 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:45 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:45.258+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:45.640 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:45 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:45.258+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:45.640 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:45 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:45.258+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:45.640 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:45 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:45.258+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-04-15T22:29:15.690645+0000 front 2024-04-15T22:29:15.690649+0000 (oldest deadline 2024-04-15T22:29:40.990639+0000) 2024-04-15T22:29:45.930 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:45 smithi176 podman[53434]: 2024-04-15 22:29:45.722417904 +0000 UTC m=+6.010464130 container cleanup 1727b9b7563530ba6cd7338f302d828e2da841eab441f1357c8a75c7ce8badfa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.buildah.version=1.19.8, release=754, vcs-type=git, version=8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, RELEASE=HEAD, architecture=x86_64, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., name=centos-stream, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-04-15T22:29:45.931 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:45 smithi176 bash[53434]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6 2024-04-15T22:29:46.208 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:45 smithi176 podman[53451]: 2024-04-15 22:29:45.931804828 +0000 UTC m=+0.589371138 container remove 1727b9b7563530ba6cd7338f302d828e2da841eab441f1357c8a75c7ce8badfa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, release=754, io.openshift.expose-services=, ceph=True, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., architecture=x86_64, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-15T22:29:46.209 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:45 smithi176 bash[53464]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T22:29:46.477 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:46 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:46.302+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:46.478 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:46 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:46.302+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:46.478 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:46 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:46.302+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:46.478 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:46 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:46.302+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-04-15T22:29:15.690645+0000 front 2024-04-15T22:29:15.690649+0000 (oldest deadline 2024-04-15T22:29:40.990639+0000) 2024-04-15T22:29:46.478 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:46 smithi176 podman[53527]: 2024-04-15 22:29:46.114162472 +0000 UTC m=+0.018835824 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:29:46.478 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:46 smithi176 podman[53527]: 2024-04-15 22:29:46.232602255 +0000 UTC m=+0.137275586 container create 962b18af3c3246b0a4e71fed1d55dc7f57e4d5157a6a9f79e0e18cdcad7cc7fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_BRANCH=HEAD, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, release=754, RELEASE=HEAD, maintainer=Guillaume Abrioux ) 2024-04-15T22:29:46.479 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:46 smithi176 podman[53527]: 2024-04-15 22:29:46.426691411 +0000 UTC m=+0.331364742 container init 962b18af3c3246b0a4e71fed1d55dc7f57e4d5157a6a9f79e0e18cdcad7cc7fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6-deactivate, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.expose-services=, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., version=8, io.openshift.tags=base centos centos-stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-15T22:29:46.479 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:46 smithi176 podman[53527]: 2024-04-15 22:29:46.430115518 +0000 UTC m=+0.334788850 container start 962b18af3c3246b0a4e71fed1d55dc7f57e4d5157a6a9f79e0e18cdcad7cc7fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6-deactivate, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.openshift.tags=base centos centos-stream, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_CLEAN=True, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, ceph=True) 2024-04-15T22:29:46.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:46 smithi176 podman[53527]: 2024-04-15 22:29:46.513707133 +0000 UTC m=+0.418380479 container attach 962b18af3c3246b0a4e71fed1d55dc7f57e4d5157a6a9f79e0e18cdcad7cc7fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6-deactivate, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, distribution-scope=public, io.openshift.expose-services=, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-15T22:29:46.978 INFO:journalctl@ceph.osd.6.smithi176.stdout:Apr 15 22:29:46 smithi176 podman[53527]: 2024-04-15 22:29:46.711609048 +0000 UTC m=+0.616282379 container died 962b18af3c3246b0a4e71fed1d55dc7f57e4d5157a6a9f79e0e18cdcad7cc7fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-6-deactivate, io.openshift.expose-services=, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, version=8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vcs-type=git, RELEASE=HEAD, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-15T22:29:47.388 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.6.service' 2024-04-15T22:29:47.432 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:29:47.432 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-15T22:29:47.432 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-15T22:29:47.432 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.7 2024-04-15T22:29:47.649 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:47 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:47.328+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:47.649 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:47 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:47.328+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:47.650 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:47 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:47.328+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:47.650 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:47 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:47.328+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-04-15T22:29:15.690645+0000 front 2024-04-15T22:29:15.690649+0000 (oldest deadline 2024-04-15T22:29:40.990639+0000) 2024-04-15T22:29:47.650 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:47 smithi176 systemd[1]: Stopping Ceph osd.7 for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:29:47.650 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:47 smithi176 bash[53604]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T22:29:47.651 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:47 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:47.650+0000 7ff8eef90700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T22:29:47.651 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:47 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:47.650+0000 7ff8eef90700 -1 osd.7 96 *** Got signal Terminated *** 2024-04-15T22:29:47.651 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:47 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:47.650+0000 7ff8eef90700 -1 osd.7 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T22:29:48.727 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:48 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:48.358+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:48.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:48 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:48.358+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:48.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:48 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:48.358+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:48.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:48 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:48.358+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-04-15T22:29:15.690645+0000 front 2024-04-15T22:29:15.690649+0000 (oldest deadline 2024-04-15T22:29:40.990639+0000) 2024-04-15T22:29:49.630 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:49 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:49.345+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:49.630 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:49 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:49.345+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:49.631 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:49 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:49.345+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:49.631 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:49 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:49.345+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-04-15T22:29:15.690645+0000 front 2024-04-15T22:29:15.690649+0000 (oldest deadline 2024-04-15T22:29:40.990639+0000) 2024-04-15T22:29:50.634 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:50 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:50.350+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:50.634 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:50 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:50.350+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:50.635 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:50 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:50.350+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:50.635 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:50 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:50.350+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-04-15T22:29:15.690645+0000 front 2024-04-15T22:29:15.690649+0000 (oldest deadline 2024-04-15T22:29:40.990639+0000) 2024-04-15T22:29:51.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:51 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:51.346+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:51.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:51 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:51.346+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:51.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:51 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:51.346+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:51.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:51 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:51.346+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-04-15T22:29:15.690645+0000 front 2024-04-15T22:29:15.690649+0000 (oldest deadline 2024-04-15T22:29:40.990639+0000) 2024-04-15T22:29:52.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:52 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:52.319+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6806 osd.0 since back 2024-04-15T22:28:52.887785+0000 front 2024-04-15T22:28:52.887747+0000 (oldest deadline 2024-04-15T22:29:15.787808+0000) 2024-04-15T22:29:52.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:52 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:52.319+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6814 osd.1 since back 2024-04-15T22:29:02.789303+0000 front 2024-04-15T22:29:02.789353+0000 (oldest deadline 2024-04-15T22:29:28.089234+0000) 2024-04-15T22:29:52.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:52 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:52.319+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6822 osd.2 since back 2024-04-15T22:29:10.390210+0000 front 2024-04-15T22:29:10.390268+0000 (oldest deadline 2024-04-15T22:29:35.690183+0000) 2024-04-15T22:29:52.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:52 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7[39239]: 2024-04-15T22:29:52.319+0000 7ff8e9927700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.92:6830 osd.3 since back 2024-04-15T22:29:15.690645+0000 front 2024-04-15T22:29:15.690649+0000 (oldest deadline 2024-04-15T22:29:40.990639+0000) 2024-04-15T22:29:53.404 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:53 smithi176 podman[53617]: 2024-04-15 22:29:53.15007246 +0000 UTC m=+5.595699469 container died 1ba050fa6fb86becc06ab4f7f91d6a1480ccff2b3085951d49cbaa7493a30306 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, release=754, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, vcs-type=git, name=centos-stream, architecture=x86_64, GIT_BRANCH=HEAD, ceph=True, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc.) 2024-04-15T22:29:53.655 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:53 smithi176 podman[53617]: 2024-04-15 22:29:53.497059468 +0000 UTC m=+5.942686477 container cleanup 1ba050fa6fb86becc06ab4f7f91d6a1480ccff2b3085951d49cbaa7493a30306 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_CLEAN=True, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, RELEASE=HEAD, architecture=x86_64, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, ceph=True, distribution-scope=public, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-15T22:29:53.655 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:53 smithi176 bash[53617]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7 2024-04-15T22:29:53.935 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:53 smithi176 podman[53675]: 2024-04-15 22:29:53.65638231 +0000 UTC m=+0.499591573 container remove 1ba050fa6fb86becc06ab4f7f91d6a1480ccff2b3085951d49cbaa7493a30306 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, version=8, io.k8s.display-name=CentOS Stream 8) 2024-04-15T22:29:53.935 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:53 smithi176 bash[53687]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T22:29:54.228 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:53 smithi176 podman[53755]: 2024-04-15 22:29:53.840351579 +0000 UTC m=+0.019203251 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T22:29:54.228 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:53 smithi176 podman[53755]: 2024-04-15 22:29:53.948952318 +0000 UTC m=+0.127804060 container create aeb90f38bf3781b56b948fc191a4fd0c5e4945eccfbe8354e73e8c7db982a3dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.openshift.expose-services=, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., architecture=x86_64, GIT_CLEAN=True, release=754, io.buildah.version=1.19.8, vcs-type=git, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-15T22:29:54.228 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:54 smithi176 podman[53755]: 2024-04-15 22:29:54.154478253 +0000 UTC m=+0.333329927 container init aeb90f38bf3781b56b948fc191a4fd0c5e4945eccfbe8354e73e8c7db982a3dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7-deactivate, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vcs-type=git, io.buildah.version=1.19.8, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, maintainer=Guillaume Abrioux , release=754, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., name=centos-stream, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, distribution-scope=public) 2024-04-15T22:29:54.228 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:54 smithi176 podman[53755]: 2024-04-15 22:29:54.157806795 +0000 UTC m=+0.336658468 container start aeb90f38bf3781b56b948fc191a4fd0c5e4945eccfbe8354e73e8c7db982a3dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7-deactivate, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, version=8, GIT_CLEAN=True, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., name=centos-stream, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git) 2024-04-15T22:29:54.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:54 smithi176 podman[53755]: 2024-04-15 22:29:54.234431709 +0000 UTC m=+0.413283391 container attach aeb90f38bf3781b56b948fc191a4fd0c5e4945eccfbe8354e73e8c7db982a3dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7-deactivate, distribution-scope=public, io.buildah.version=1.19.8, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, com.redhat.component=centos-stream-container, vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, vendor=Red Hat, Inc., ceph=True, architecture=x86_64, GIT_BRANCH=HEAD, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-15T22:29:54.728 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:54 smithi176 podman[53755]: 2024-04-15 22:29:54.466434309 +0000 UTC m=+0.645285985 container died aeb90f38bf3781b56b948fc191a4fd0c5e4945eccfbe8354e73e8c7db982a3dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7-deactivate, vcs-type=git, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, distribution-scope=public, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-15T22:29:55.014 INFO:journalctl@ceph.osd.7.smithi176.stdout:Apr 15 22:29:55 smithi176 podman[53857]: 2024-04-15 22:29:55.012631778 +0000 UTC m=+0.537194609 container remove aeb90f38bf3781b56b948fc191a4fd0c5e4945eccfbe8354e73e8c7db982a3dc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-osd-7-deactivate, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, com.redhat.component=centos-stream-container, architecture=x86_64, io.openshift.expose-services=, vcs-type=git, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, version=8, name=centos-stream, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-15T22:29:55.029 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@osd.7.service' 2024-04-15T22:29:55.074 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:29:55.074 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-15T22:29:55.074 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-15T22:29:55.075 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ceph-d537646e-fb75-11ee-bc8f-c7b262605968@prometheus.a 2024-04-15T22:29:55.289 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 systemd[1]: Stopping Ceph prometheus.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:29:55.289 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 bash[53954]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus.a" found: no such container 2024-04-15T22:29:55.676 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:29:55.290Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-15T22:29:55.676 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:29:55.290Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-15T22:29:55.676 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:29:55.290Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-15T22:29:55.677 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:29:55.290Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-15T22:29:55.677 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:29:55.290Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-15T22:29:55.677 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:29:55.290Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-15T22:29:55.677 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:29:55.290Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-15T22:29:55.677 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:29:55.290Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-15T22:29:55.677 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:29:55.290Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-15T22:29:55.677 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:29:55.291Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-15T22:29:55.677 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:29:55.291Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-15T22:29:55.678 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a[45673]: ts=2024-04-15T22:29:55.291Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-15T22:29:55.678 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 podman[53967]: 2024-04-15 22:29:55.301158348 +0000 UTC m=+0.111745959 container died 4c2f0c2b64ee94a74ba0409b65531c02eb5cefda674fdb267494acfd9d1000f7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:29:55.931 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 podman[53967]: 2024-04-15 22:29:55.698710432 +0000 UTC m=+0.509298044 container cleanup 4c2f0c2b64ee94a74ba0409b65531c02eb5cefda674fdb267494acfd9d1000f7 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T22:29:55.932 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 bash[53967]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus-a 2024-04-15T22:29:55.932 INFO:journalctl@ceph.prometheus.a.smithi176.stdout:Apr 15 22:29:55 smithi176 bash[54011]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-prometheus.a" found: no such container 2024-04-15T22:29:55.936 DEBUG:teuthology.orchestra.run.smithi176:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d537646e-fb75-11ee-bc8f-c7b262605968@prometheus.a.service' 2024-04-15T22:29:55.976 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T22:29:55.976 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-15T22:29:55.976 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d537646e-fb75-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-15T22:29:58.963 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:29:58 smithi092 systemd[1]: Stopping Ceph node-exporter.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:29:58.963 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:29:58 smithi092 bash[65329]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter.a" found: no such container 2024-04-15T22:29:58.963 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:29:58 smithi092 podman[65342]: 2024-04-15 22:29:58.772135034 +0000 UTC m=+0.120284825 container died 2b352622d41eb20126c2c102fbd09e9e826531a8e44c23ae4fd4a3dc72b08017 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-15T22:29:59.318 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:29:59 smithi092 podman[65342]: 2024-04-15 22:29:59.136062433 +0000 UTC m=+0.484212231 container cleanup 2b352622d41eb20126c2c102fbd09e9e826531a8e44c23ae4fd4a3dc72b08017 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-15T22:29:59.319 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:29:59 smithi092 bash[65342]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a 2024-04-15T22:29:59.672 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:29:59 smithi092 systemd[1]: Stopping Ceph alertmanager.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:29:59.672 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:29:59 smithi092 bash[65421]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager.a" found: no such container 2024-04-15T22:29:59.673 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:29:59 smithi092 podman[65356]: 2024-04-15 22:29:59.32038085 +0000 UTC m=+0.552321509 container remove 2b352622d41eb20126c2c102fbd09e9e826531a8e44c23ae4fd4a3dc72b08017 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-15T22:29:59.673 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:29:59 smithi092 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-15T22:29:59.673 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:29:59 smithi092 bash[65369]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter.a" found: no such container 2024-04-15T22:29:59.673 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:29:59 smithi092 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-15T22:29:59.673 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:29:59 smithi092 systemd[1]: Stopped Ceph node-exporter.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:29:59.673 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Apr 15 22:29:59 smithi092 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@node-exporter.a.service: Consumed 3.408s CPU time. 2024-04-15T22:29:59.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:29:59 smithi092 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a[51750]: level=info ts=2024-04-15T22:29:59.673Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-15T22:29:59.963 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:29:59 smithi092 podman[65434]: 2024-04-15 22:29:59.704388811 +0000 UTC m=+0.136513308 container died 5d76ee1dbbdb2155e8edef2cd4ab4b6731288b897844695e133a26f36c27a507 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T22:30:00.315 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:30:00 smithi092 podman[65434]: 2024-04-15 22:30:00.065524647 +0000 UTC m=+0.497649141 container cleanup 5d76ee1dbbdb2155e8edef2cd4ab4b6731288b897844695e133a26f36c27a507 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T22:30:00.315 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:30:00 smithi092 bash[65434]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a 2024-04-15T22:30:00.315 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:30:00 smithi092 bash[65460]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager.a" found: no such container 2024-04-15T22:30:00.586 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:30:00 smithi092 podman[65448]: 2024-04-15 22:30:00.316542586 +0000 UTC m=+0.613849088 container remove 5d76ee1dbbdb2155e8edef2cd4ab4b6731288b897844695e133a26f36c27a507 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T22:30:00.586 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:30:00 smithi092 podman[65448]: 2024-04-15 22:30:00.391967855 +0000 UTC m=+0.689274356 volume remove 8037f3a6b0ee4fb5b0489e078d3c4e25a5aea58aba006d8152a47b804e795a84 2024-04-15T22:30:00.587 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:30:00 smithi092 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-15T22:30:00.587 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Apr 15 22:30:00 smithi092 systemd[1]: Stopped Ceph alertmanager.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:30:34.782 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d537646e-fb75-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-15T22:30:36.622 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:30:36 smithi176 systemd[1]: Stopping Ceph node-exporter.b for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:30:36.622 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:30:36 smithi176 bash[55120]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter.b" found: no such container 2024-04-15T22:30:36.903 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:30:36 smithi176 podman[55132]: 2024-04-15 22:30:36.623158115 +0000 UTC m=+0.113226775 container died 774c4598b14ff40483d2860ca79091ee0904043353c9db21596788c5f5f81b01 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-15T22:30:37.155 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:30:36 smithi176 podman[55132]: 2024-04-15 22:30:36.978815097 +0000 UTC m=+0.468883757 container cleanup 774c4598b14ff40483d2860ca79091ee0904043353c9db21596788c5f5f81b01 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-15T22:30:37.155 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:30:36 smithi176 bash[55132]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b 2024-04-15T22:30:37.155 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:30:37 smithi176 podman[55146]: 2024-04-15 22:30:37.154775543 +0000 UTC m=+0.534415327 container remove 774c4598b14ff40483d2860ca79091ee0904043353c9db21596788c5f5f81b01 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-15T22:30:37.408 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:30:37 smithi176 systemd[1]: Stopping Ceph grafana.a for d537646e-fb75-11ee-bc8f-c7b262605968... 2024-04-15T22:30:37.409 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:30:37 smithi176 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-15T22:30:37.409 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:30:37 smithi176 bash[55159]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-node-exporter.b" found: no such container 2024-04-15T22:30:37.409 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:30:37 smithi176 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-15T22:30:37.409 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:30:37 smithi176 systemd[1]: Stopped Ceph node-exporter.b for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:30:37.409 INFO:journalctl@ceph.node-exporter.b.smithi176.stdout:Apr 15 22:30:37 smithi176 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@node-exporter.b.service: Consumed 3.397s CPU time. 2024-04-15T22:30:37.724 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:30:37 smithi176 bash[55211]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana.a" found: no such container 2024-04-15T22:30:37.724 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:30:37 smithi176 ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a[44653]: t=2024-04-15T22:30:37+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-04-15T22:30:37.725 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:30:37 smithi176 podman[55222]: 2024-04-15 22:30:37.518416382 +0000 UTC m=+0.100538235 container died 1233e94881e9603abe96e431fd04326e6163d4488f47816024b9ba4ebd7f7389 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a, distribution-scope=public, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, build-date=2022-03-28T10:36:18.413762, name=ubi8, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, maintainer=Paul Cuzner , description=Ceph Grafana Container, io.openshift.expose-services=, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.24.2) 2024-04-15T22:30:37.977 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:30:37 smithi176 podman[55222]: 2024-04-15 22:30:37.807806703 +0000 UTC m=+0.389928544 container cleanup 1233e94881e9603abe96e431fd04326e6163d4488f47816024b9ba4ebd7f7389 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, com.redhat.component=ubi8-container, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , distribution-scope=public, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-04-15T22:30:37.978 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:30:37 smithi176 bash[55222]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a 2024-04-15T22:30:37.978 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:30:37 smithi176 bash[55251]: Error: no container with name or ID "ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana.a" found: no such container 2024-04-15T22:30:38.352 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:30:38 smithi176 podman[55236]: 2024-04-15 22:30:38.025419704 +0000 UTC m=+0.507338697 container remove 1233e94881e9603abe96e431fd04326e6163d4488f47816024b9ba4ebd7f7389 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-d537646e-fb75-11ee-bc8f-c7b262605968-grafana-a, architecture=x86_64, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, distribution-scope=public, io.openshift.expose-services=, io.openshift.tags=base rhel8, name=ubi8, vcs-type=git, maintainer=Paul Cuzner , io.buildah.version=1.24.2, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-04-15T22:30:38.352 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:30:38 smithi176 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-15T22:30:38.352 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:30:38 smithi176 systemd[1]: Stopped Ceph grafana.a for d537646e-fb75-11ee-bc8f-c7b262605968. 2024-04-15T22:30:38.352 INFO:journalctl@ceph.grafana.a.smithi176.stdout:Apr 15 22:30:38 smithi176 systemd[1]: ceph-d537646e-fb75-11ee-bc8f-c7b262605968@grafana.a.service: Consumed 2.774s CPU time. 2024-04-15T22:31:00.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:31:00.461 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T22:31:00.492 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-15T22:31:00.494 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657551/remote/smithi092/crash 2024-04-15T22:31:00.495 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/crash -- . 2024-04-15T22:31:00.546 INFO:teuthology.orchestra.run.smithi092.stderr:tar: /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-15T22:31:00.546 INFO:teuthology.orchestra.run.smithi092.stderr:tar: Error is not recoverable: exiting now 2024-04-15T22:31:00.548 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657551/remote/smithi176/crash 2024-04-15T22:31:00.549 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/crash -- . 2024-04-15T22:31:00.589 INFO:teuthology.orchestra.run.smithi176.stderr:tar: /var/lib/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-15T22:31:00.590 INFO:teuthology.orchestra.run.smithi176.stderr:tar: Error is not recoverable: exiting now 2024-04-15T22:31:00.591 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-15T22:31:00.591 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-15T22:31:00.624 INFO:teuthology.orchestra.run.smithi092.stdout:2024-04-15T22:19:03.811675+0000 mon.a (mon.0) 199 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-15T22:31:00.624 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-15T22:31:00.625 INFO:tasks.cephadm:Compressing logs... 2024-04-15T22:31:00.625 DEBUG:teuthology.orchestra.run.smithi092:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T22:31:00.666 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T22:31:00.693 INFO:teuthology.orchestra.run.smithi092.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-15T22:31:00.693 INFO:teuthology.orchestra.run.smithi092.stderr:: No such file or directory 2024-04-15T22:31:00.694 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mon.a.log 2024-04-15T22:31:00.695 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose --/var/log/ceph/cephadm.log: /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-client.rgw.foo.smithi092.xmypxq.log 2024-04-15T22:31:00.695 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-15T22:31:00.695 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-client.rgw.foo.smithi092.xmypxq.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.1.log 2024-04-15T22:31:00.695 INFO:teuthology.orchestra.run.smithi092.stderr: 62.4% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-client.rgw.foo.smithi092.xmypxq.log.gz 2024-04-15T22:31:00.696 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.3.log 2024-04-15T22:31:00.708 INFO:teuthology.orchestra.run.smithi176.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-15T22:31:00.708 INFO:teuthology.orchestra.run.smithi176.stderr:: No such file or directory 2024-04-15T22:31:00.708 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.6.log 2024-04-15T22:31:00.708 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose --/var/log/ceph/cephadm.log: /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-client.rgw.smpl.smithi176.fwngry.log 2024-04-15T22:31:00.708 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-15T22:31:00.708 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-client.rgw.smpl.smithi176.fwngry.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.4.log 2024-04-15T22:31:00.709 INFO:teuthology.orchestra.run.smithi176.stderr: 62.4% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-client.rgw.smpl.smithi176.fwngry.log.gz 2024-04-15T22:31:00.709 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.7.log 2024-04-15T22:31:00.709 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.5.log 2024-04-15T22:31:00.709 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.4.log: /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mgr.x.log 2024-04-15T22:31:00.709 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-client.rgw.foo.smithi176.twznrb.log 2024-04-15T22:31:00.709 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-15T22:31:00.709 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-client.rgw.foo.smithi176.twznrb.log: 62.4% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-client.rgw.foo.smithi176.twznrb.log.gz 90.3% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-15T22:31:00.709 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-04-15T22:31:00.710 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.log 2024-04-15T22:31:00.710 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-volume.log 81.4% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-15T22:31:00.710 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-04-15T22:31:00.710 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mon.b.log 2024-04-15T22:31:00.710 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-volume.log: /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.log: 87.5% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-15T22:31:00.710 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mon.b.log: 91.2% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mgr.x.log.gz 2024-04-15T22:31:00.710 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.audit.log: /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-client.rgw.smpl.smithi092.juhhhh.log 2024-04-15T22:31:00.711 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mgr.y.log 2024-04-15T22:31:00.711 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-client.rgw.smpl.smithi092.juhhhh.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/tcmu-runner.log 2024-04-15T22:31:00.711 INFO:teuthology.orchestra.run.smithi092.stderr: 62.4% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-client.rgw.smpl.smithi092.juhhhh.log.gz 2024-04-15T22:31:00.711 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.0.log 2024-04-15T22:31:00.711 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-15T22:31:00.711 INFO:teuthology.orchestra.run.smithi092.stderr: 66.7% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/tcmu-runner.log.gz 2024-04-15T22:31:00.711 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.log 2024-04-15T22:31:00.711 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.cephadm.log: 89.2% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-15T22:31:00.712 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.2.log 2024-04-15T22:31:00.712 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.log: 94.0% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-15T22:31:00.712 INFO:teuthology.orchestra.run.smithi092.stderr: 92.7% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-15T22:31:00.712 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-15T22:31:00.712 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mon.c.log 2024-04-15T22:31:00.712 INFO:teuthology.orchestra.run.smithi176.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-15T22:31:00.714 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-volume.log: 91.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-15T22:31:00.720 INFO:teuthology.orchestra.run.smithi176.stderr: 92.4% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-15T22:31:00.726 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mon.c.log: 92.4% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-15T22:31:00.850 INFO:teuthology.orchestra.run.smithi176.stderr: 92.2% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mon.b.log.gz 2024-04-15T22:31:00.899 INFO:teuthology.orchestra.run.smithi092.stderr: 87.6% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mgr.y.log.gz 2024-04-15T22:31:00.961 INFO:teuthology.orchestra.run.smithi092.stderr: 92.1% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mon.c.log.gz 2024-04-15T22:31:01.028 INFO:teuthology.orchestra.run.smithi092.stderr: 93.4% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.2.log.gz 2024-04-15T22:31:01.086 INFO:teuthology.orchestra.run.smithi176.stderr: 93.2% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.6.log.gz 2024-04-15T22:31:01.097 INFO:teuthology.orchestra.run.smithi176.stderr: 93.9% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.7.log.gz 2024-04-15T22:31:01.140 INFO:teuthology.orchestra.run.smithi176.stderr: 93.7% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.4.log.gz 2024-04-15T22:31:01.176 INFO:teuthology.orchestra.run.smithi176.stderr: 93.5% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.5.log.gz 2024-04-15T22:31:01.178 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-04-15T22:31:01.178 INFO:teuthology.orchestra.run.smithi176.stderr:real 0m0.497s 2024-04-15T22:31:01.179 INFO:teuthology.orchestra.run.smithi176.stderr:user 0m1.847s 2024-04-15T22:31:01.179 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m0.097s 2024-04-15T22:31:01.207 INFO:teuthology.orchestra.run.smithi092.stderr: 93.6% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.1.log.gz 2024-04-15T22:31:01.227 INFO:teuthology.orchestra.run.smithi092.stderr: 93.5% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.0.log.gz 2024-04-15T22:31:01.304 INFO:teuthology.orchestra.run.smithi092.stderr: 91.1% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-mon.a.log.gz 2024-04-15T22:31:01.335 INFO:teuthology.orchestra.run.smithi092.stderr: 93.6% -- replaced with /var/log/ceph/d537646e-fb75-11ee-bc8f-c7b262605968/ceph-osd.3.log.gz 2024-04-15T22:31:01.337 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-04-15T22:31:01.337 INFO:teuthology.orchestra.run.smithi092.stderr:real 0m0.657s 2024-04-15T22:31:01.337 INFO:teuthology.orchestra.run.smithi092.stderr:user 0m3.000s 2024-04-15T22:31:01.337 INFO:teuthology.orchestra.run.smithi092.stderr:sys 0m0.131s 2024-04-15T22:31:01.337 INFO:tasks.cephadm:Archiving logs... 2024-04-15T22:31:01.338 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657551/remote/smithi092/log 2024-04-15T22:31:01.339 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T22:31:01.765 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657551/remote/smithi176/log 2024-04-15T22:31:01.766 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T22:31:02.054 INFO:tasks.cephadm:Removing cluster... 2024-04-15T22:31:02.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d537646e-fb75-11ee-bc8f-c7b262605968 --force 2024-04-15T22:31:02.330 DEBUG:teuthology.orchestra.run.smithi176:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d537646e-fb75-11ee-bc8f-c7b262605968 --force 2024-04-15T22:31:02.609 INFO:tasks.cephadm:Removing cephadm ... 2024-04-15T22:31:02.610 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-15T22:31:02.626 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-15T22:31:02.641 INFO:tasks.cephadm:Teardown complete 2024-04-15T22:31:02.642 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-15T22:31:02.652 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-15T22:31:02.652 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T22:31:02.668 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T22:31:02.680 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-04-15T22:31:02.685 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T22:31:02.685 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-04-15T22:31:02.685 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv01.front.sepia.ceph.com 3 6 377 14 +933us[ +938us] +/- 48ms 2024-04-15T22:31:02.685 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 16 -456us[ -452us] +/- 57ms 2024-04-15T22:31:02.685 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 -888us[ -888us] +/- 72ms 2024-04-15T22:31:02.685 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:31:02.695 INFO:teuthology.orchestra.run.smithi176.stderr:bash: line 1: ntpq: command not found 2024-04-15T22:31:02.700 INFO:teuthology.orchestra.run.smithi176.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T22:31:02.700 INFO:teuthology.orchestra.run.smithi176.stdout:=============================================================================== 2024-04-15T22:31:02.700 INFO:teuthology.orchestra.run.smithi176.stdout:^* hv01.front.sepia.ceph.com 3 6 377 16 +950us[ +952us] +/- 48ms 2024-04-15T22:31:02.700 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 15 -442us[ -442us] +/- 57ms 2024-04-15T22:31:02.700 INFO:teuthology.orchestra.run.smithi176.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 -876us[ -876us] +/- 72ms 2024-04-15T22:31:02.700 INFO:teuthology.orchestra.run.smithi176.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T22:31:02.701 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-15T22:31:02.743 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-15T22:31:02.744 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-15T22:31:02.789 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T22:31:02.867 DEBUG:teuthology.orchestra.run.smithi176:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T22:31:02.938 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-15T22:31:02.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:31:02.970 DEBUG:teuthology.orchestra.run.smithi176:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-15T22:31:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:31:03.002 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-15T22:31:03.011 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-15T22:31:03.063 INFO:teuthology.task.internal:Duration was 1133.039437 seconds 2024-04-15T22:31:03.063 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-15T22:31:03.096 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-15T22:31:03.096 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T22:31:03.099 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T22:31:03.142 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T22:31:03.144 INFO:teuthology.orchestra.run.smithi176.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T22:31:03.498 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-15T22:31:03.498 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2024-04-15T22:31:03.498 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T22:31:03.518 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-04-15T22:31:03.519 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T22:31:03.542 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-15T22:31:03.543 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T22:31:03.561 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T22:31:03.612 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-15T22:31:03.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T22:31:03.631 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T22:31:03.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-15T22:31:03.964 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-15T22:31:03.965 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T22:31:03.994 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T22:31:04.024 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-15T22:31:04.034 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T22:31:04.037 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T22:31:04.061 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2024-04-15T22:31:04.091 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-04-15T22:31:04.112 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T22:31:04.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:31:04.141 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T22:31:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T22:31:04.168 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-15T22:31:04.178 INFO:teuthology.task.internal:Transferring archived files... 2024-04-15T22:31:04.179 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657551/remote/smithi092 2024-04-15T22:31:04.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T22:31:04.231 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657551/remote/smithi176 2024-04-15T22:31:04.231 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T22:31:04.273 INFO:teuthology.task.internal:Removing archive directory... 2024-04-15T22:31:04.274 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T22:31:04.277 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T22:31:04.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-15T22:31:04.335 INFO:teuthology.task.internal:Not uploading archives. 2024-04-15T22:31:04.335 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-15T22:31:04.382 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-15T22:31:04.383 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T22:31:04.386 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T22:31:04.400 INFO:teuthology.orchestra.run.smithi092.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 22:31 /home/ubuntu/cephtest 2024-04-15T22:31:04.401 INFO:teuthology.orchestra.run.smithi176.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 22:31 /home/ubuntu/cephtest 2024-04-15T22:31:04.402 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-15T22:31:04.420 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-15T22:31:04.440 INFO:teuthology.nuke:Checking targets against current locks 2024-04-15T22:31:04.472 DEBUG:teuthology.nuke:shortname: smithi092 2024-04-15T22:31:04.472 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T22:31:04.495 DEBUG:teuthology.nuke:shortname: smithi176 2024-04-15T22:31:04.495 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T22:31:04.500 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657551', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 22:01:20.270610', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCPI1D4wbeFTYbyUH82+GuJSNRt54o9VFq15ChczDGJZ/bsLn4/BLY20WoTa6eLTd8Pxp4+1Pcfcri2pTKQz7CU='} 2024-04-15T22:31:04.526 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657551', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 22:01:20.269350', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGQnmaIcuCYUx6jXt9aYBab0fCtYtGH6H5NO5Z7BKyKIMsi8Tnm7Qaxg+kBz0lEDsAKEUJYnyZBpgAyU0bHMrVI='} 2024-04-15T22:31:04.554 INFO:teuthology.orchestra.console.smithi092:Power off 2024-04-15T22:31:04.555 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T22:31:04.579 DEBUG:teuthology.orchestra.console.smithi092:power off output: Chassis Power Control: Down/Off 2024-04-15T22:31:04.579 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T22:31:04.597 INFO:teuthology.orchestra.console.smithi176:Power off 2024-04-15T22:31:04.597 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T22:31:04.608 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-04-15T22:31:04.621 DEBUG:teuthology.orchestra.console.smithi176:power off output: Chassis Power Control: Down/Off 2024-04-15T22:31:04.621 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T22:31:04.646 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-04-15T22:31:08.609 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T22:31:08.647 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T22:31:08.719 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-04-15T22:31:08.758 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is on 2024-04-15T22:31:12.725 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T22:31:12.759 DEBUG:teuthology.orchestra.console.smithi176:pexpect command: ipmitool -H smithi176.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T22:31:12.834 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is off 2024-04-15T22:31:12.869 DEBUG:teuthology.orchestra.console.smithi176:check power output: Chassis Power is off 2024-04-15T22:31:12.936 INFO:teuthology.orchestra.console.smithi092:Power off completed 2024-04-15T22:31:12.971 INFO:teuthology.orchestra.console.smithi176:Power off completed 2024-04-15T22:31:13.076 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} duration: 1133.0394368171692 failure_reason: 'Command failed on smithi092 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d537646e-fb75-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c \''ceph versions | jq -e \''"\''"\''.overall | keys\''"\''"\'' | grep $sha1\''''' owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4835b59349454ef2b34618d3e60df80c status: fail success: false 2024-04-15T22:31:13.077 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T22:31:13.152 INFO:teuthology.run:FAIL