2024-04-22T21:56:11.523 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-22T21:56:11.523 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668811 --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} --verbose --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668811/orig.config.yaml 2024-04-22T21:56:11.553 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T21:56:11.755 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668811 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} email: ceph-qa@ceph.com first_in_suite: false job_id: '7668811' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 1 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: true mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-only-match: - CEPHADM_ sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f workunit: branch: squid sha1: c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 207 sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f sleep_before_teardown: 0 subset: 24/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9 targets: smithi174.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLCmdAO+N68U0PLDLxlMe57+huFXk4h+ektZmUJNIBph9/kp1iHAT/aAZr6HprXbgEaQVy3XhXELyOlZ0BQvL+I= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP1XXqiS8lqW9G9jAODYdBqySY2EUZPuRew3OjNPF0/ZR2ZC9/IZu84f4Q8SZ1CmLfuoNDRpeGlrfuoXlECwxqI= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: mon.a: - ceph config set mgr mgr/cephadm/use_repo_digest false --force - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-22_21:08:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-22T21:56:11.756 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa; will attempt to use it 2024-04-22T21:56:11.756 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks 2024-04-22T21:56:11.756 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-22T21:56:11.758 INFO:teuthology.task.internal:Checking packages... 2024-04-22T21:56:11.781 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '430e09df97c8fc7dc2b2ae424f68ed11366c540f' 2024-04-22T21:56:11.781 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-22T21:56:11.781 INFO:teuthology.packaging:ref: None 2024-04-22T21:56:11.781 INFO:teuthology.packaging:tag: None 2024-04-22T21:56:11.781 INFO:teuthology.packaging:branch: squid 2024-04-22T21:56:11.781 INFO:teuthology.packaging:sha1: 430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T21:56:11.781 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-22T21:56:11.996 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2420.g246d7127 2024-04-22T21:56:11.997 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-22T21:56:12.004 INFO:teuthology.task.internal:no buildpackages task found 2024-04-22T21:56:12.004 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-22T21:56:12.009 INFO:teuthology.task.internal:Saving configuration 2024-04-22T21:56:12.020 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-22T21:56:12.026 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T21:56:12.050 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668811', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 21:50:16.053059', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLCmdAO+N68U0PLDLxlMe57+huFXk4h+ektZmUJNIBph9/kp1iHAT/aAZr6HprXbgEaQVy3XhXELyOlZ0BQvL+I='} 2024-04-22T21:56:12.072 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668811', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 21:50:16.051497', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP1XXqiS8lqW9G9jAODYdBqySY2EUZPuRew3OjNPF0/ZR2ZC9/IZu84f4Q8SZ1CmLfuoNDRpeGlrfuoXlECwxqI='} 2024-04-22T21:56:12.072 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-22T21:56:12.079 INFO:teuthology.task.internal:roles: ubuntu@smithi174.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-04-22T21:56:12.079 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-04-22T21:56:12.079 INFO:teuthology.run_tasks:Running task console_log... 2024-04-22T21:56:12.143 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fecab7660d0>, signals=[15]) 2024-04-22T21:56:12.144 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-22T21:56:12.150 INFO:teuthology.task.internal:Opening connections... 2024-04-22T21:56:12.150 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi174.front.sepia.ceph.com 2024-04-22T21:56:12.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T21:56:12.227 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-04-22T21:56:12.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T21:56:12.302 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-22T21:56:12.308 DEBUG:teuthology.orchestra.run.smithi174:> uname -m 2024-04-22T21:56:12.326 INFO:teuthology.orchestra.run.smithi174.stdout:x86_64 2024-04-22T21:56:12.326 DEBUG:teuthology.orchestra.run.smithi174:> cat /etc/os-release 2024-04-22T21:56:12.383 INFO:teuthology.orchestra.run.smithi174.stdout:NAME="CentOS Stream" 2024-04-22T21:56:12.383 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION="9" 2024-04-22T21:56:12.383 INFO:teuthology.orchestra.run.smithi174.stdout:ID="centos" 2024-04-22T21:56:12.383 INFO:teuthology.orchestra.run.smithi174.stdout:ID_LIKE="rhel fedora" 2024-04-22T21:56:12.383 INFO:teuthology.orchestra.run.smithi174.stdout:VERSION_ID="9" 2024-04-22T21:56:12.383 INFO:teuthology.orchestra.run.smithi174.stdout:PLATFORM_ID="platform:el9" 2024-04-22T21:56:12.384 INFO:teuthology.orchestra.run.smithi174.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T21:56:12.384 INFO:teuthology.orchestra.run.smithi174.stdout:ANSI_COLOR="0;31" 2024-04-22T21:56:12.384 INFO:teuthology.orchestra.run.smithi174.stdout:LOGO="fedora-logo-icon" 2024-04-22T21:56:12.384 INFO:teuthology.orchestra.run.smithi174.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T21:56:12.384 INFO:teuthology.orchestra.run.smithi174.stdout:HOME_URL="https://centos.org/" 2024-04-22T21:56:12.384 INFO:teuthology.orchestra.run.smithi174.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T21:56:12.384 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T21:56:12.384 INFO:teuthology.orchestra.run.smithi174.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T21:56:12.385 INFO:teuthology.lock.ops:Updating smithi174.front.sepia.ceph.com on lock server 2024-04-22T21:56:12.516 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-04-22T21:56:12.534 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-04-22T21:56:12.534 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-04-22T21:56:12.592 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-04-22T21:56:12.592 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="9" 2024-04-22T21:56:12.592 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-04-22T21:56:12.593 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-04-22T21:56:12.593 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="9" 2024-04-22T21:56:12.593 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el9" 2024-04-22T21:56:12.593 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-22T21:56:12.593 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-04-22T21:56:12.593 INFO:teuthology.orchestra.run.smithi192.stdout:LOGO="fedora-logo-icon" 2024-04-22T21:56:12.593 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-22T21:56:12.593 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-04-22T21:56:12.593 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-22T21:56:12.593 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-22T21:56:12.593 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-22T21:56:12.593 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-04-22T21:56:12.613 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-22T21:56:12.623 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-22T21:56:12.628 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-22T21:56:12.629 DEBUG:teuthology.orchestra.run.smithi174:> test '!' -e /home/ubuntu/cephtest 2024-04-22T21:56:12.631 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-04-22T21:56:12.649 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-22T21:56:12.654 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-22T21:56:12.654 DEBUG:teuthology.orchestra.run.smithi174:> test -z $(ls -A /var/lib/ceph) 2024-04-22T21:56:12.688 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-04-22T21:56:12.801 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-22T21:56:12.854 INFO:teuthology.run_tasks:Running task kernel... 2024-04-22T21:56:12.868 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-22T21:56:12.869 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-22T21:56:12.869 DEBUG:teuthology.orchestra.run.smithi174:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T21:56:12.870 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-22T21:56:12.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:56:12.886 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-04-22T21:56:12.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:56:12.887 DEBUG:teuthology.orchestra.run.smithi174:> uname -r 2024-04-22T21:56:12.943 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-437.el9.x86_64 2024-04-22T21:56:12.943 INFO:teuthology.task.kernel:Running kernel on smithi192: 5.14.0-437.el9.x86_64 2024-04-22T21:56:12.943 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2024-04-22T21:56:12.945 INFO:teuthology.orchestra.run.smithi174.stdout:5.14.0-437.el9.x86_64 2024-04-22T21:56:12.945 INFO:teuthology.task.kernel:Running kernel on smithi174: 5.14.0-437.el9.x86_64 2024-04-22T21:56:12.945 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum install -y kernel 2024-04-22T21:56:15.141 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-04-22T21:56:16.160 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.0 MB 00:01 2024-04-22T21:56:18.061 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-04-22T21:56:22.464 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 6.2 MB 00:00 2024-04-22T21:56:24.137 INFO:teuthology.orchestra.run.smithi174.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-04-22T21:56:25.088 INFO:teuthology.orchestra.run.smithi174.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-04-22T21:56:29.757 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 1.4 MB/s | 19 MB 00:13 2024-04-22T21:56:30.163 INFO:teuthology.orchestra.run.smithi174.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-04-22T21:56:32.061 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T21:56:32.061 INFO:teuthology.orchestra.run.smithi174.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T21:56:32.150 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-04-22T21:56:32.151 INFO:teuthology.orchestra.run.smithi174.stdout:Nothing to do. 2024-04-22T21:56:32.151 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-04-22T21:56:32.275 DEBUG:teuthology.orchestra.run.smithi174:> echo no | sudo yum reinstall kernel || true 2024-04-22T21:56:32.779 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:02 ago on Mon 22 Apr 2024 09:56:30 PM UTC. 2024-04-22T21:56:32.899 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-04-22T21:56:32.917 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-04-22T21:56:32.917 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-04-22T21:56:32.918 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-04-22T21:56:32.918 INFO:teuthology.orchestra.run.smithi174.stdout:Reinstalling: 2024-04-22T21:56:32.918 INFO:teuthology.orchestra.run.smithi174.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T21:56:32.918 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T21:56:32.918 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-04-22T21:56:32.918 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-04-22T21:56:32.918 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T21:56:32.918 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 255 k 2024-04-22T21:56:32.918 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 0 2024-04-22T21:56:32.918 INFO:teuthology.orchestra.run.smithi174.stderr:Operation aborted. 2024-04-22T21:56:32.977 INFO:teuthology.orchestra.run.smithi174.stdout:Is this ok [y/N]: 2024-04-22T21:56:32.977 DEBUG:teuthology.orchestra.run.smithi174:> sudo yum reinstall -y kernel || true 2024-04-22T21:56:33.468 INFO:teuthology.orchestra.run.smithi174.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 09:56:30 PM UTC. 2024-04-22T21:56:33.584 INFO:teuthology.orchestra.run.smithi174.stdout:Dependencies resolved. 2024-04-22T21:56:33.585 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-04-22T21:56:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: Package Architecture Version Repository Size 2024-04-22T21:56:33.585 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-04-22T21:56:33.585 INFO:teuthology.orchestra.run.smithi174.stdout:Reinstalling: 2024-04-22T21:56:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T21:56:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T21:56:33.585 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction Summary 2024-04-22T21:56:33.585 INFO:teuthology.orchestra.run.smithi174.stdout:================================================================================ 2024-04-22T21:56:33.585 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T21:56:33.585 INFO:teuthology.orchestra.run.smithi174.stdout:Total download size: 255 k 2024-04-22T21:56:33.585 INFO:teuthology.orchestra.run.smithi174.stdout:Installed size: 0 2024-04-22T21:56:33.585 INFO:teuthology.orchestra.run.smithi174.stdout:Downloading Packages: 2024-04-22T21:56:34.332 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-437.el9.x86_64.rpm 531 kB/s | 255 kB 00:00 2024-04-22T21:56:34.332 INFO:teuthology.orchestra.run.smithi174.stdout:-------------------------------------------------------------------------------- 2024-04-22T21:56:34.332 INFO:teuthology.orchestra.run.smithi174.stdout:Total 341 kB/s | 255 kB 00:00 2024-04-22T21:56:34.349 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction check 2024-04-22T21:56:34.543 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction check succeeded. 2024-04-22T21:56:34.543 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction test 2024-04-22T21:56:34.548 INFO:teuthology.orchestra.run.smithi174.stdout:Transaction test succeeded. 2024-04-22T21:56:34.548 INFO:teuthology.orchestra.run.smithi174.stdout:Running transaction 2024-04-22T21:56:34.683 INFO:teuthology.orchestra.run.smithi174.stdout: Preparing : 1/1 2024-04-22T21:56:34.775 INFO:teuthology.orchestra.run.smithi174.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T21:56:34.775 INFO:teuthology.orchestra.run.smithi174.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:56:35.216 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.2 MB 00:01 2024-04-22T21:56:35.501 INFO:teuthology.orchestra.run.smithi174.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:56:35.501 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T21:56:36.204 INFO:teuthology.orchestra.run.smithi174.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:56:36.204 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T21:56:36.204 INFO:teuthology.orchestra.run.smithi174.stdout:Reinstalled: 2024-04-22T21:56:36.205 INFO:teuthology.orchestra.run.smithi174.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T21:56:36.205 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T21:56:36.205 INFO:teuthology.orchestra.run.smithi174.stdout:Complete! 2024-04-22T21:56:36.370 DEBUG:teuthology.orchestra.run.smithi174:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T21:56:36.417 INFO:teuthology.orchestra.run.smithi174.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T21:56:36.418 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T21:56:36.418 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T21:56:36.418 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-22T21:56:36.418 DEBUG:teuthology.orchestra.run.smithi174:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T21:56:36.447 INFO:teuthology.orchestra.run.smithi174.stdout:ttyS1 2024-04-22T21:56:36.467 DEBUG:teuthology.parallel:result is None 2024-04-22T21:56:36.833 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-04-22T21:56:38.110 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-04-22T21:56:43.192 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-04-22T21:56:44.999 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-22T21:56:44.999 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-04-22T21:56:45.104 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-22T21:56:45.105 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-04-22T21:56:45.105 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-22T21:56:45.217 DEBUG:teuthology.orchestra.run.smithi192:> echo no | sudo yum reinstall kernel || true 2024-04-22T21:56:45.718 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:02 ago on Mon 22 Apr 2024 09:56:43 PM UTC. 2024-04-22T21:56:45.845 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-22T21:56:45.856 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-22T21:56:45.856 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-22T21:56:45.857 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-22T21:56:45.857 INFO:teuthology.orchestra.run.smithi192.stdout:Reinstalling: 2024-04-22T21:56:45.857 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T21:56:45.857 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-22T21:56:45.857 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-22T21:56:45.857 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-22T21:56:45.857 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-22T21:56:45.858 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 255 k 2024-04-22T21:56:45.858 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 0 2024-04-22T21:56:45.858 INFO:teuthology.orchestra.run.smithi192.stderr:Operation aborted. 2024-04-22T21:56:45.919 INFO:teuthology.orchestra.run.smithi192.stdout:Is this ok [y/N]: 2024-04-22T21:56:45.919 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum reinstall -y kernel || true 2024-04-22T21:56:46.415 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:03 ago on Mon 22 Apr 2024 09:56:43 PM UTC. 2024-04-22T21:56:46.532 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-04-22T21:56:46.533 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-22T21:56:46.533 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-04-22T21:56:46.533 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-22T21:56:46.533 INFO:teuthology.orchestra.run.smithi192.stdout:Reinstalling: 2024-04-22T21:56:46.533 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-04-22T21:56:46.533 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-22T21:56:46.533 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-04-22T21:56:46.533 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-04-22T21:56:46.533 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-22T21:56:46.533 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 255 k 2024-04-22T21:56:46.533 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 0 2024-04-22T21:56:46.533 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-04-22T21:56:47.286 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-437.el9.x86_64.rpm 418 kB/s | 255 kB 00:00 2024-04-22T21:56:47.286 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-04-22T21:56:47.286 INFO:teuthology.orchestra.run.smithi192.stdout:Total 338 kB/s | 255 kB 00:00 2024-04-22T21:56:47.301 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-04-22T21:56:47.470 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-04-22T21:56:47.471 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-04-22T21:56:47.475 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-04-22T21:56:47.476 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-04-22T21:56:47.622 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-04-22T21:56:47.695 INFO:teuthology.orchestra.run.smithi192.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T21:56:47.696 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:56:48.337 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:56:48.337 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-04-22T21:56:49.041 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-04-22T21:56:49.041 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-22T21:56:49.041 INFO:teuthology.orchestra.run.smithi192.stdout:Reinstalled: 2024-04-22T21:56:49.041 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-5.14.0-437.el9.x86_64 2024-04-22T21:56:49.041 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-22T21:56:49.041 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-04-22T21:56:49.219 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-04-22T21:56:49.265 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-437.el9.x86_64 2024-04-22T21:56:49.265 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-04-22T21:56:49.266 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-04-22T21:56:49.266 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-22T21:56:49.266 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-22T21:56:49.295 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2024-04-22T21:56:49.320 DEBUG:teuthology.parallel:result is None 2024-04-22T21:56:49.321 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-22T21:56:49.328 INFO:teuthology.task.internal:Creating test directory... 2024-04-22T21:56:49.328 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T21:56:49.331 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-22T21:56:49.376 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-22T21:56:49.383 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-22T21:56:49.393 INFO:teuthology.task.internal:Creating archive directory... 2024-04-22T21:56:49.393 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T21:56:49.395 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-22T21:56:49.443 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-22T21:56:49.449 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-22T21:56:49.449 DEBUG:teuthology.orchestra.run.smithi174:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T21:56:49.478 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-22T21:56:49.506 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T21:56:49.516 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T21:56:49.522 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T21:56:49.533 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-22T21:56:49.535 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-22T21:56:49.542 INFO:teuthology.task.internal:Configuring sudo... 2024-04-22T21:56:49.543 DEBUG:teuthology.orchestra.run.smithi174:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T21:56:49.545 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-22T21:56:49.583 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-22T21:56:49.594 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-22T21:56:49.594 DEBUG:teuthology.orchestra.run.smithi174:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T21:56:49.612 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-22T21:56:49.637 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T21:56:49.708 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T21:56:49.755 DEBUG:teuthology.orchestra.run.smithi174:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T21:56:49.834 DEBUG:teuthology.orchestra.run.smithi174:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T21:56:49.878 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-22T21:56:49.878 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T21:56:49.942 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T21:56:49.986 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-22T21:56:50.038 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T21:56:50.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-22T21:56:50.158 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-22T21:56:50.158 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-22T21:56:50.221 DEBUG:teuthology.orchestra.run.smithi174:> sudo service rsyslog restart 2024-04-22T21:56:50.223 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-04-22T21:56:50.270 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T21:56:50.298 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T21:56:50.607 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-22T21:56:50.615 INFO:teuthology.task.internal:Starting timer... 2024-04-22T21:56:50.615 INFO:teuthology.run_tasks:Running task pcp... 2024-04-22T21:56:50.636 INFO:teuthology.run_tasks:Running task selinux... 2024-04-22T21:56:50.644 DEBUG:teuthology.orchestra.run.smithi174:> sudo service auditd rotate 2024-04-22T21:56:50.703 INFO:teuthology.orchestra.run.smithi174.stdout:Rotating logs: 2024-04-22T21:56:50.706 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-04-22T21:56:50.746 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-04-22T21:56:50.748 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-22T21:56:50.748 DEBUG:teuthology.orchestra.run.smithi174:> /usr/sbin/getenforce 2024-04-22T21:56:50.775 INFO:teuthology.orchestra.run.smithi174.stdout:Permissive 2024-04-22T21:56:50.775 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-04-22T21:56:50.810 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-04-22T21:56:50.810 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi174.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-04-22T21:56:50.810 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T21:56:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:56:50.841 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T21:56:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T21:56:50.874 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-22T21:56:50.874 DEBUG:teuthology.orchestra.run.smithi174:> sudo /usr/sbin/setenforce permissive 2024-04-22T21:56:50.906 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-04-22T21:56:50.939 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-22T21:56:50.949 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-22T21:56:50.965 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-22T21:56:50.966 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi174.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-22T22:00:22.618 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi174.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-04-22T22:00:22.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-04-22T22:00:22.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi174.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T22:00:22.699 DEBUG:teuthology.orchestra.run.smithi174:> true 2024-04-22T22:00:22.779 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi174.front.sepia.ceph.com' 2024-04-22T22:00:22.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-22T22:00:22.780 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-22T22:00:22.862 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-04-22T22:00:22.942 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-04-22T22:00:22.942 INFO:teuthology.run_tasks:Running task clock... 2024-04-22T22:00:22.954 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-22T22:00:22.954 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T22:00:22.955 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T22:00:22.957 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-22T22:00:22.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T22:00:22.982 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T22:00:22.996 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T22:00:23.011 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-22T22:00:23.019 INFO:teuthology.orchestra.run.smithi174.stderr:sudo: ntpd: command not found 2024-04-22T22:00:23.024 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-22T22:00:23.030 INFO:teuthology.orchestra.run.smithi174.stdout:506 Cannot talk to daemon 2024-04-22T22:00:23.042 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T22:00:23.047 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-04-22T22:00:23.055 INFO:teuthology.orchestra.run.smithi174.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T22:00:23.057 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-04-22T22:00:23.070 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-22T22:00:23.083 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-22T22:00:23.106 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-04-22T22:00:23.110 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T22:00:23.110 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-04-22T22:00:23.110 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:00:23.110 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:00:23.110 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:00:23.110 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:00:23.135 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-04-22T22:00:23.138 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T22:00:23.138 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-04-22T22:00:23.138 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:00:23.138 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:00:23.138 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:00:23.138 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:00:23.139 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-22T22:00:23.242 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 1}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': True}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-only-match': ['CEPHADM_'], 'sha1': '430e09df97c8fc7dc2b2ae424f68ed11366c540f'} 2024-04-22T22:00:23.242 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-22T22:00:23.242 INFO:tasks.cephadm:Cluster fsid is b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:00:23.242 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-22T22:00:23.243 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.174', 'mon.c': '[v2:172.21.15.174:3301,v1:172.21.15.174:6790]', 'mon.b': '172.21.15.192'} 2024-04-22T22:00:23.243 INFO:tasks.cephadm:First mon is mon.a on smithi174 2024-04-22T22:00:23.243 INFO:tasks.cephadm:First mgr is y 2024-04-22T22:00:23.243 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-22T22:00:23.243 DEBUG:teuthology.orchestra.run.smithi174:> sudo hostname $(hostname -s) 2024-04-22T22:00:23.267 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2024-04-22T22:00:23.292 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-22T22:00:23.293 DEBUG:teuthology.orchestra.run.smithi174:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T22:00:23.547 INFO:teuthology.orchestra.run.smithi174.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 22 22:00 /home/ubuntu/cephtest/cephadm 2024-04-22T22:00:23.548 DEBUG:teuthology.orchestra.run.smithi192:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-22T22:00:23.730 INFO:teuthology.orchestra.run.smithi192.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 22 22:00 /home/ubuntu/cephtest/cephadm 2024-04-22T22:00:23.731 DEBUG:teuthology.orchestra.run.smithi174:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T22:00:23.746 DEBUG:teuthology.orchestra.run.smithi192:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-22T22:00:23.777 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-22T22:00:23.777 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-22T22:00:23.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-22T22:00:25.298 INFO:teuthology.orchestra.run.smithi174.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-22T22:00:25.330 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-22T22:00:48.032 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-04-22T22:00:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-22T22:00:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-22T22:00:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2024-04-22T22:00:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-22T22:00:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-22T22:00:48.032 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-04-22T22:00:48.032 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-04-22T22:00:48.237 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-22T22:00:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-22T22:00:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-22T22:00:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "repo_digests": [ 2024-04-22T22:00:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-22T22:00:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-22T22:00:48.237 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-04-22T22:00:48.237 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-22T22:00:48.255 DEBUG:teuthology.orchestra.run.smithi174:> sudo mkdir -p /etc/ceph 2024-04-22T22:00:48.291 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2024-04-22T22:00:48.318 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 777 /etc/ceph 2024-04-22T22:00:48.354 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2024-04-22T22:00:48.381 INFO:tasks.cephadm:Writing seed config... 2024-04-22T22:00:48.382 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-04-22T22:00:48.382 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-22T22:00:48.382 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-22T22:00:48.382 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = True 2024-04-22T22:00:48.383 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-22T22:00:48.383 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-22T22:00:48.383 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-22T22:00:48.383 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-22T22:00:48.383 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-22T22:00:48.383 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-22T22:00:48.383 DEBUG:teuthology.orchestra.run.smithi174:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-22T22:00:48.409 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = b7cbfdbc-00f3-11ef-bc93-c7b262605968 mon election default strategy = 1 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = True [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-22T22:00:48.410 DEBUG:teuthology.orchestra.run.smithi174:mon.a> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mon.a.service 2024-04-22T22:00:48.453 DEBUG:teuthology.orchestra.run.smithi174:mgr.y> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mgr.y.service 2024-04-22T22:00:48.495 INFO:tasks.cephadm:Bootstrapping... 2024-04-22T22:00:48.496 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.174 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:00:48.685 INFO:teuthology.orchestra.run.smithi174.stderr:-------------------------------------------------------------------------------- 2024-04-22T22:00:48.685 INFO:teuthology.orchestra.run.smithi174.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', 'b7cbfdbc-00f3-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.174', '--skip-admin-label'] 2024-04-22T22:00:48.708 INFO:teuthology.orchestra.run.smithi174.stderr:/bin/podman: 4.9.4-dev 2024-04-22T22:00:48.710 INFO:teuthology.orchestra.run.smithi174.stderr:Verifying podman|docker is present... 2024-04-22T22:00:48.732 INFO:teuthology.orchestra.run.smithi174.stderr:/bin/podman: 4.9.4-dev 2024-04-22T22:00:48.734 INFO:teuthology.orchestra.run.smithi174.stderr:Verifying lvm2 is present... 2024-04-22T22:00:48.734 INFO:teuthology.orchestra.run.smithi174.stderr:Verifying time synchronization is in place... 2024-04-22T22:00:48.741 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T22:00:48.747 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: inactive 2024-04-22T22:00:48.754 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: enabled 2024-04-22T22:00:48.760 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: active 2024-04-22T22:00:48.760 INFO:teuthology.orchestra.run.smithi174.stderr:Unit chronyd.service is enabled and running 2024-04-22T22:00:48.760 INFO:teuthology.orchestra.run.smithi174.stderr:Repeating the final host check... 2024-04-22T22:00:48.781 INFO:teuthology.orchestra.run.smithi174.stderr:/bin/podman: 4.9.4-dev 2024-04-22T22:00:48.783 INFO:teuthology.orchestra.run.smithi174.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-22T22:00:48.783 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl is present 2024-04-22T22:00:48.783 INFO:teuthology.orchestra.run.smithi174.stderr:lvcreate is present 2024-04-22T22:00:48.789 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-22T22:00:48.795 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: inactive 2024-04-22T22:00:48.801 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: enabled 2024-04-22T22:00:48.807 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: active 2024-04-22T22:00:48.809 INFO:teuthology.orchestra.run.smithi174.stderr:Unit chronyd.service is enabled and running 2024-04-22T22:00:48.809 INFO:teuthology.orchestra.run.smithi174.stderr:Host looks OK 2024-04-22T22:00:48.809 INFO:teuthology.orchestra.run.smithi174.stderr:Cluster fsid: b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:00:48.809 INFO:teuthology.orchestra.run.smithi174.stderr:Acquiring lock 140634156728176 on /run/cephadm/b7cbfdbc-00f3-11ef-bc93-c7b262605968.lock 2024-04-22T22:00:48.809 INFO:teuthology.orchestra.run.smithi174.stderr:Lock 140634156728176 acquired on /run/cephadm/b7cbfdbc-00f3-11ef-bc93-c7b262605968.lock 2024-04-22T22:00:48.809 INFO:teuthology.orchestra.run.smithi174.stderr:Verifying IP 172.21.15.174 port 3300 ... 2024-04-22T22:00:48.809 INFO:teuthology.orchestra.run.smithi174.stderr:Verifying IP 172.21.15.174 port 6789 ... 2024-04-22T22:00:48.809 INFO:teuthology.orchestra.run.smithi174.stderr:Base mon IP is 172.21.15.174, final addrv is [v2:172.21.15.174:3300,v1:172.21.15.174:6789] 2024-04-22T22:00:48.811 INFO:teuthology.orchestra.run.smithi174.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.174 metric 100 2024-04-22T22:00:48.811 INFO:teuthology.orchestra.run.smithi174.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.174 metric 100 2024-04-22T22:00:48.814 INFO:teuthology.orchestra.run.smithi174.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-22T22:00:48.814 INFO:teuthology.orchestra.run.smithi174.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-04-22T22:00:48.814 INFO:teuthology.orchestra.run.smithi174.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-04-22T22:00:48.816 INFO:teuthology.orchestra.run.smithi174.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-22T22:00:48.817 INFO:teuthology.orchestra.run.smithi174.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-22T22:00:48.817 INFO:teuthology.orchestra.run.smithi174.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-22T22:00:48.817 INFO:teuthology.orchestra.run.smithi174.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-22T22:00:48.817 INFO:teuthology.orchestra.run.smithi174.stderr:/sbin/ip: inet6 fe80::80ab:e611:e75e:5e32/64 scope link noprefixroute 2024-04-22T22:00:48.817 INFO:teuthology.orchestra.run.smithi174.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-22T22:00:48.817 INFO:teuthology.orchestra.run.smithi174.stderr:Mon IP `172.21.15.174` is in CIDR network `172.21.0.0/20` 2024-04-22T22:00:48.817 INFO:teuthology.orchestra.run.smithi174.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-22T22:00:48.818 INFO:teuthology.orchestra.run.smithi174.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-22T22:00:48.919 INFO:teuthology.orchestra.run.smithi174.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-22T22:00:49.572 INFO:teuthology.orchestra.run.smithi174.stderr:/bin/podman: Getting image source signatures 2024-04-22T22:00:49.572 INFO:teuthology.orchestra.run.smithi174.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-22T22:00:49.572 INFO:teuthology.orchestra.run.smithi174.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-22T22:00:49.572 INFO:teuthology.orchestra.run.smithi174.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-22T22:00:49.572 INFO:teuthology.orchestra.run.smithi174.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-22T22:00:49.572 INFO:teuthology.orchestra.run.smithi174.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-22T22:00:49.572 INFO:teuthology.orchestra.run.smithi174.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-22T22:00:49.708 INFO:teuthology.orchestra.run.smithi174.stderr:/bin/podman: Writing manifest to image destination 2024-04-22T22:00:49.822 INFO:teuthology.orchestra.run.smithi174.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-22T22:00:50.248 INFO:teuthology.orchestra.run.smithi174.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-22T22:00:50.752 INFO:teuthology.orchestra.run.smithi174.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-22T22:00:50.752 INFO:teuthology.orchestra.run.smithi174.stderr:Extracting ceph user uid/gid from container image... 2024-04-22T22:00:51.468 INFO:teuthology.orchestra.run.smithi174.stderr:stat: 167 167 2024-04-22T22:00:54.851 INFO:teuthology.orchestra.run.smithi174.stderr:Creating initial keys... 2024-04-22T22:00:58.562 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph-authtool: AQAZ3iZml3/xMBAAZyvKlSQwEANxykC6Xtl5BQ== 2024-04-22T22:01:01.947 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph-authtool: AQAd3iZmnu1iNBAAm0/AJbXNBSKhivtQCCYTtQ== 2024-04-22T22:01:03.419 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph-authtool: AQAf3iZm7oZdFRAAZEhoN9oHkiFGRVB/W8LV3Q== 2024-04-22T22:01:03.984 INFO:teuthology.orchestra.run.smithi174.stderr:Creating initial monmap... 2024-04-22T22:01:04.382 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T22:01:04.383 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-22T22:01:04.383 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:01:04.383 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T22:01:04.988 INFO:teuthology.orchestra.run.smithi174.stderr:monmaptool for a [v2:172.21.15.174:3300,v1:172.21.15.174:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-22T22:01:04.989 INFO:teuthology.orchestra.run.smithi174.stderr:setting min_mon_release = octopus 2024-04-22T22:01:04.989 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/monmaptool: set fsid to b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:01:04.989 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-22T22:01:04.989 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-22T22:01:04.989 INFO:teuthology.orchestra.run.smithi174.stderr:Creating mon... 2024-04-22T22:01:05.961 INFO:teuthology.orchestra.run.smithi174.stderr:create mon.a on 2024-04-22T22:01:06.174 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-22T22:01:06.325 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968.target. 2024-04-22T22:01:06.325 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968.target. 2024-04-22T22:01:06.679 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: Failed to reset failed state of unit ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-22T22:01:06.686 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: Created symlink /etc/systemd/system/ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968.target.wants/ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@.service. 2024-04-22T22:01:07.240 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:07 smithi174 podman[21445]: 2024-04-22 22:01:07.078595553 +0000 UTC m=+0.018509780 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-22T22:01:07.240 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:07 smithi174 podman[21445]: 2024-04-22 22:01:07.26218741 +0000 UTC m=+0.202101617 container create 6d51b6f9cdbf6c0ec1a4de5fec19ee7a6a52c59ae4d8e5722f52d4527e033ed0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, distribution-scope=public, release=754, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8) 2024-04-22T22:01:07.491 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:07 smithi174 podman[21445]: 2024-04-22 22:01:07.456064129 +0000 UTC m=+0.395978338 container init 6d51b6f9cdbf6c0ec1a4de5fec19ee7a6a52c59ae4d8e5722f52d4527e033ed0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, release=754, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, version=8, GIT_CLEAN=True, com.redhat.component=centos-stream-container, architecture=x86_64, distribution-scope=public, io.buildah.version=1.19.8) 2024-04-22T22:01:07.491 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:07 smithi174 podman[21445]: 2024-04-22 22:01:07.459112306 +0000 UTC m=+0.399026527 container start 6d51b6f9cdbf6c0ec1a4de5fec19ee7a6a52c59ae4d8e5722f52d4527e033ed0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, ceph=True, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., architecture=x86_64, RELEASE=HEAD, io.buildah.version=1.19.8, vcs-type=git) 2024-04-22T22:01:07.491 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:07 smithi174 ceph-mon[21465]: mkfs b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:01:07.491 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:07 smithi174 ceph-mon[21465]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T22:01:07.491 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:07 smithi174 bash[21445]: 6d51b6f9cdbf6c0ec1a4de5fec19ee7a6a52c59ae4d8e5722f52d4527e033ed0 2024-04-22T22:01:07.504 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: disabled 2024-04-22T22:01:07.511 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: inactive 2024-04-22T22:01:07.512 INFO:teuthology.orchestra.run.smithi174.stderr:firewalld.service is not enabled 2024-04-22T22:01:07.513 INFO:teuthology.orchestra.run.smithi174.stderr:Not possible to enable service . firewalld.service is not available 2024-04-22T22:01:07.513 INFO:teuthology.orchestra.run.smithi174.stderr:Waiting for mon to start... 2024-04-22T22:01:07.513 INFO:teuthology.orchestra.run.smithi174.stderr:Waiting for mon... 2024-04-22T22:01:07.750 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:07 smithi174 systemd[1]: Started Ceph mon.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:01:08.045 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: cluster: 2024-04-22T22:01:08.045 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: id: b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:01:08.045 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-22T22:01:08.045 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 2024-04-22T22:01:08.046 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: services: 2024-04-22T22:01:08.046 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.572293s) 2024-04-22T22:01:08.046 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-22T22:01:08.046 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-22T22:01:08.046 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 2024-04-22T22:01:08.046 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: data: 2024-04-22T22:01:08.046 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-22T22:01:08.046 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-22T22:01:08.046 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-22T22:01:08.046 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: pgs: 2024-04-22T22:01:08.046 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 2024-04-22T22:01:08.565 INFO:teuthology.orchestra.run.smithi174.stderr:mon is available 2024-04-22T22:01:08.565 INFO:teuthology.orchestra.run.smithi174.stderr:Assimilating anything we can from ceph.conf... 2024-04-22T22:01:08.689 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:08 smithi174 ceph-mon[21465]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T22:01:08.689 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:08 smithi174 ceph-mon[21465]: monmap e1: 1 mons at {a=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0]} 2024-04-22T22:01:08.690 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:08 smithi174 ceph-mon[21465]: fsmap 2024-04-22T22:01:08.690 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:08 smithi174 ceph-mon[21465]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T22:01:08.690 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:08 smithi174 ceph-mon[21465]: mgrmap e1: no daemons active 2024-04-22T22:01:08.690 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:08 smithi174 ceph-mon[21465]: from='client.? 172.21.15.174:0/678988942' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-22T22:01:09.081 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 2024-04-22T22:01:09.081 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: [global] 2024-04-22T22:01:09.082 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: fsid = b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:01:09.082 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.174:3300,v1:172.21.15.174:6789] 2024-04-22T22:01:09.082 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-22T22:01:09.082 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-22T22:01:09.082 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-22T22:01:09.083 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-22T22:01:09.083 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 2024-04-22T22:01:09.083 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: [mgr] 2024-04-22T22:01:09.083 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = True 2024-04-22T22:01:09.083 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-22T22:01:09.083 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 2024-04-22T22:01:09.083 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: [osd] 2024-04-22T22:01:09.084 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-22T22:01:09.084 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-22T22:01:09.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:09 smithi174 ceph-mon[21465]: from='client.? 172.21.15.174:0/4179631584' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T22:01:09.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:09 smithi174 ceph-mon[21465]: from='client.? 172.21.15.174:0/4179631584' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T22:01:09.655 INFO:teuthology.orchestra.run.smithi174.stderr:Generating new minimal ceph.conf... 2024-04-22T22:01:10.800 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:10 smithi174 ceph-mon[21465]: from='client.? 172.21.15.174:0/1771930199' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:01:10.804 INFO:teuthology.orchestra.run.smithi174.stderr:Restarting the monitor... 2024-04-22T22:01:11.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:10 smithi174 systemd[1]: Stopping Ceph mon.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:01:11.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:10 smithi174 bash[21695]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon.a" found: no such container 2024-04-22T22:01:11.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:10 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a[21459]: 2024-04-22T22:01:10.975+0000 7fd19dcf5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:01:11.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:10 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a[21459]: 2024-04-22T22:01:10.975+0000 7fd19dcf5700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-22T22:01:11.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 podman[21707]: 2024-04-22 22:01:11.017779723 +0000 UTC m=+0.136398538 container died 6d51b6f9cdbf6c0ec1a4de5fec19ee7a6a52c59ae4d8e5722f52d4527e033ed0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, release=754, ceph=True, name=centos-stream, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, RELEASE=HEAD, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vcs-type=git, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64) 2024-04-22T22:01:11.530 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 podman[21707]: 2024-04-22 22:01:11.351052885 +0000 UTC m=+0.469671709 container cleanup 6d51b6f9cdbf6c0ec1a4de5fec19ee7a6a52c59ae4d8e5722f52d4527e033ed0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, vendor=Red Hat, Inc., architecture=x86_64, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-22T22:01:11.531 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 bash[21707]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a 2024-04-22T22:01:11.531 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 podman[21721]: 2024-04-22 22:01:11.53523316 +0000 UTC m=+0.518338539 container remove 6d51b6f9cdbf6c0ec1a4de5fec19ee7a6a52c59ae4d8e5722f52d4527e033ed0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a, version=8, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, name=centos-stream, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, release=754, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_CLEAN=True, ceph=True, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-22T22:01:11.531 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 bash[21735]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon.a" found: no such container 2024-04-22T22:01:11.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 systemd[1]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-22T22:01:11.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 systemd[1]: Stopped Ceph mon.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:01:11.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 systemd[1]: Starting Ceph mon.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:01:11.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 podman[21807]: 2024-04-22 22:01:11.777582101 +0000 UTC m=+0.092515823 container create d85a5efa7dc5763848bb269686567df898f06fd442d4ff43a9b6d16923597215 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , distribution-scope=public, io.buildah.version=1.19.8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, version=8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., name=centos-stream, io.k8s.display-name=CentOS Stream 8) 2024-04-22T22:01:11.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 podman[21807]: 2024-04-22 22:01:11.703681615 +0000 UTC m=+0.018615349 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-22T22:01:11.992 INFO:teuthology.orchestra.run.smithi174.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-22T22:01:12.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 podman[21807]: 2024-04-22 22:01:11.946942491 +0000 UTC m=+0.261876225 container init d85a5efa7dc5763848bb269686567df898f06fd442d4ff43a9b6d16923597215 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, vcs-type=git, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.openshift.expose-services=, distribution-scope=public) 2024-04-22T22:01:12.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 podman[21807]: 2024-04-22 22:01:11.950244433 +0000 UTC m=+0.265178155 container start d85a5efa7dc5763848bb269686567df898f06fd442d4ff43a9b6d16923597215 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, com.redhat.component=centos-stream-container, distribution-scope=public, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, release=754, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0) 2024-04-22T22:01:12.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T22:01:12.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-22T22:01:12.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: pidfile_write: ignore empty --pid-file 2024-04-22T22:01:12.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: load: jerasure load: lrc 2024-04-22T22:01:12.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: RocksDB version: 6.15.5 2024-04-22T22:01:12.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-22T22:01:12.108 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Compile date Apr 18 2022 2024-04-22T22:01:12.108 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: DB SUMMARY 2024-04-22T22:01:12.108 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: DB Session ID: DH3BNUAMEO8OL0ER1HZ1 2024-04-22T22:01:12.108 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: CURRENT file: CURRENT 2024-04-22T22:01:12.108 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T22:01:12.108 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-22T22:01:12.108 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-22T22:01:12.108 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82659 ; 2024-04-22T22:01:12.108 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.error_if_exists: 0 2024-04-22T22:01:12.108 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.create_if_missing: 0 2024-04-22T22:01:12.108 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.paranoid_checks: 1 2024-04-22T22:01:12.108 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T22:01:12.108 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.env: 0x55839a0c6860 2024-04-22T22:01:12.109 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.fs: Posix File System 2024-04-22T22:01:12.109 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.info_log: 0x55839bc1bdc0 2024-04-22T22:01:12.109 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T22:01:12.109 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.statistics: (nil) 2024-04-22T22:01:12.109 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.use_fsync: 0 2024-04-22T22:01:12.109 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_log_file_size: 0 2024-04-22T22:01:12.109 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T22:01:12.109 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T22:01:12.109 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T22:01:12.109 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T22:01:12.109 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.allow_fallocate: 1 2024-04-22T22:01:12.109 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T22:01:12.109 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T22:01:12.110 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.use_direct_reads: 0 2024-04-22T22:01:12.110 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T22:01:12.110 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T22:01:12.110 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.db_log_dir: 2024-04-22T22:01:12.110 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-04-22T22:01:12.110 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T22:01:12.110 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T22:01:12.110 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T22:01:12.111 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T22:01:12.111 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T22:01:12.111 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T22:01:12.111 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T22:01:12.113 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T22:01:12.113 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.write_buffer_manager: 0x55839bd0c240 2024-04-22T22:01:12.114 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T22:01:12.114 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-22T22:01:12.114 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T22:01:12.114 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T22:01:12.114 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T22:01:12.114 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T22:01:12.114 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T22:01:12.115 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T22:01:12.115 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T22:01:12.115 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.unordered_write: 0 2024-04-22T22:01:12.115 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T22:01:12.115 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T22:01:12.115 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T22:01:12.115 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T22:01:12.116 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.row_cache: None 2024-04-22T22:01:12.116 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.wal_filter: None 2024-04-22T22:01:12.116 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T22:01:12.116 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T22:01:12.116 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.preserve_deletes: 0 2024-04-22T22:01:12.116 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.two_write_queues: 0 2024-04-22T22:01:12.116 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T22:01:12.116 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.atomic_flush: 0 2024-04-22T22:01:12.117 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T22:01:12.117 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T22:01:12.117 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T22:01:12.117 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.log_readahead_size: 0 2024-04-22T22:01:12.117 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T22:01:12.117 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T22:01:12.117 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T22:01:12.117 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T22:01:12.117 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T22:01:12.118 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T22:01:12.118 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_background_jobs: 2 2024-04-22T22:01:12.118 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_background_compactions: -1 2024-04-22T22:01:12.118 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_subcompactions: 1 2024-04-22T22:01:12.118 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T22:01:12.118 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T22:01:12.119 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T22:01:12.119 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T22:01:12.119 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T22:01:12.119 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T22:01:12.119 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T22:01:12.119 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T22:01:12.119 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_open_files: -1 2024-04-22T22:01:12.120 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T22:01:12.120 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T22:01:12.120 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T22:01:12.120 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T22:01:12.120 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_background_flushes: -1 2024-04-22T22:01:12.120 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Compression algorithms supported: 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: kZSTD supported: 0 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: kXpressCompression supported: 0 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: kLZ4Compression supported: 1 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: kBZip2Compression supported: 0 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: kZlibCompression supported: 1 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: kSnappyCompression supported: 1 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.merge_operator: 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compaction_filter: None 2024-04-22T22:01:12.121 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T22:01:12.122 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T22:01:12.122 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T22:01:12.122 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T22:01:12.122 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55839bbe7d00) 2024-04-22T22:01:12.122 INFO:journalctl@ceph.mon.a.smithi174.stdout: cache_index_and_filter_blocks: 1 2024-04-22T22:01:12.122 INFO:journalctl@ceph.mon.a.smithi174.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T22:01:12.122 INFO:journalctl@ceph.mon.a.smithi174.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T22:01:12.122 INFO:journalctl@ceph.mon.a.smithi174.stdout: pin_top_level_index_and_filter: 1 2024-04-22T22:01:12.122 INFO:journalctl@ceph.mon.a.smithi174.stdout: index_type: 0 2024-04-22T22:01:12.122 INFO:journalctl@ceph.mon.a.smithi174.stdout: data_block_index_type: 0 2024-04-22T22:01:12.122 INFO:journalctl@ceph.mon.a.smithi174.stdout: index_shortening: 1 2024-04-22T22:01:12.122 INFO:journalctl@ceph.mon.a.smithi174.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T22:01:12.122 INFO:journalctl@ceph.mon.a.smithi174.stdout: hash_index_allow_collision: 1 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: checksum: 1 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: no_block_cache: 0 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: block_cache: 0x55839bc52170 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: block_cache_name: BinnedLRUCache 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: block_cache_options: 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: capacity : 536870912 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: num_shard_bits : 4 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: strict_capacity_limit : 0 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: high_pri_pool_ratio: 0.000 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: block_cache_compressed: (nil) 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: persistent_cache: (nil) 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: block_size: 4096 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: block_size_deviation: 10 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: block_restart_interval: 16 2024-04-22T22:01:12.123 INFO:journalctl@ceph.mon.a.smithi174.stdout: index_block_restart_interval: 1 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout: metadata_block_size: 4096 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout: partition_filters: 0 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout: use_delta_encoding: 1 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout: whole_key_filtering: 1 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout: verify_compression: 0 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout: read_amp_bytes_per_bit: 0 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout: format_version: 4 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout: enable_index_compression: 1 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout: block_align: 0 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compression: NoCompression 2024-04-22T22:01:12.124 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T22:01:12.125 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T22:01:12.125 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T22:01:12.125 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.num_levels: 7 2024-04-22T22:01:12.125 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T22:01:12.125 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T22:01:12.125 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T22:01:12.125 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T22:01:12.125 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T22:01:12.125 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T22:01:12.125 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T22:01:12.125 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T22:01:12.125 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T22:01:12.125 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T22:01:12.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T22:01:12.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T22:01:12.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T22:01:12.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T22:01:12.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T22:01:12.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T22:01:12.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T22:01:12.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T22:01:12.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T22:01:12.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T22:01:12.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T22:01:12.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T22:01:12.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T22:01:12.127 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T22:01:12.127 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T22:01:12.127 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T22:01:12.127 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T22:01:12.127 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T22:01:12.127 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T22:01:12.127 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T22:01:12.127 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T22:01:12.127 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T22:01:12.127 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T22:01:12.128 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T22:01:12.128 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.arena_block_size: 4194304 2024-04-22T22:01:12.128 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T22:01:12.128 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T22:01:12.128 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-22T22:01:12.128 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T22:01:12.128 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T22:01:12.128 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T22:01:12.128 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T22:01:12.128 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T22:01:12.128 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T22:01:12.129 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T22:01:12.129 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T22:01:12.129 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T22:01:12.129 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T22:01:12.129 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T22:01:12.129 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.table_properties_collectors: 2024-04-22T22:01:12.129 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.inplace_update_support: 0 2024-04-22T22:01:12.129 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T22:01:12.130 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T22:01:12.130 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T22:01:12.130 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T22:01:12.130 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.bloom_locality: 0 2024-04-22T22:01:12.130 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.max_successive_merges: 0 2024-04-22T22:01:12.130 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T22:01:12.130 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T22:01:12.131 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T22:01:12.131 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T22:01:12.131 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.ttl: 2592000 2024-04-22T22:01:12.131 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T22:01:12.131 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.enable_blob_files: false 2024-04-22T22:01:12.131 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.min_blob_size: 0 2024-04-22T22:01:12.131 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T22:01:12.132 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T22:01:12.132 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T22:01:12.132 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T22:01:12.132 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T22:01:12.132 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-22T22:01:12.132 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-22T22:01:12.133 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713823271976136, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-22T22:01:12.133 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-22T22:01:12.133 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-22T22:01:12.133 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713823271977130, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79209, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77472, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71781, "raw_average_value_size": 340, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713823271, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "cba20ff1-25ac-4fdc-8d7a-6b32912be124", "db_session_id": "DH3BNUAMEO8OL0ER1HZ1"}} 2024-04-22T22:01:12.133 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-22T22:01:12.133 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713823271977595, "job": 1, "event": "recovery_finished"} 2024-04-22T22:01:12.133 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T22:01:12.134 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55839bc38a80 2024-04-22T22:01:12.134 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: DB pointer 0x55839bc48000 2024-04-22T22:01:12.134 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-22T22:01:12.134 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-22T22:01:12.134 INFO:journalctl@ceph.mon.a.smithi174.stdout: ** DB Stats ** 2024-04-22T22:01:12.134 INFO:journalctl@ceph.mon.a.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:01:12.134 INFO:journalctl@ceph.mon.a.smithi174.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T22:01:12.134 INFO:journalctl@ceph.mon.a.smithi174.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T22:01:12.135 INFO:journalctl@ceph.mon.a.smithi174.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T22:01:12.135 INFO:journalctl@ceph.mon.a.smithi174.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T22:01:12.135 INFO:journalctl@ceph.mon.a.smithi174.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-22T22:01:12.135 INFO:journalctl@ceph.mon.a.smithi174.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T22:01:12.135 INFO:journalctl@ceph.mon.a.smithi174.stdout: 2024-04-22T22:01:12.135 INFO:journalctl@ceph.mon.a.smithi174.stdout: ** Compaction Stats [default] ** 2024-04-22T22:01:12.135 INFO:journalctl@ceph.mon.a.smithi174.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:01:12.136 INFO:journalctl@ceph.mon.a.smithi174.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:01:12.136 INFO:journalctl@ceph.mon.a.smithi174.stdout: L0 2/0 79.11 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.8 0.00 0.00 1 0.001 0 0 2024-04-22T22:01:12.136 INFO:journalctl@ceph.mon.a.smithi174.stdout: Sum 2/0 79.11 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.8 0.00 0.00 1 0.001 0 0 2024-04-22T22:01:12.136 INFO:journalctl@ceph.mon.a.smithi174.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.8 0.00 0.00 1 0.001 0 0 2024-04-22T22:01:12.136 INFO:journalctl@ceph.mon.a.smithi174.stdout: 2024-04-22T22:01:12.136 INFO:journalctl@ceph.mon.a.smithi174.stdout: ** Compaction Stats [default] ** 2024-04-22T22:01:12.136 INFO:journalctl@ceph.mon.a.smithi174.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:01:12.137 INFO:journalctl@ceph.mon.a.smithi174.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:01:12.137 INFO:journalctl@ceph.mon.a.smithi174.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 114.8 0.00 0.00 1 0.001 0 0 2024-04-22T22:01:12.137 INFO:journalctl@ceph.mon.a.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:01:12.137 INFO:journalctl@ceph.mon.a.smithi174.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T22:01:12.137 INFO:journalctl@ceph.mon.a.smithi174.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T22:01:12.137 INFO:journalctl@ceph.mon.a.smithi174.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T22:01:12.137 INFO:journalctl@ceph.mon.a.smithi174.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T22:01:12.137 INFO:journalctl@ceph.mon.a.smithi174.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T22:01:12.138 INFO:journalctl@ceph.mon.a.smithi174.stdout: Cumulative compaction: 0.00 GB write, 18.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:01:12.138 INFO:journalctl@ceph.mon.a.smithi174.stdout: Interval compaction: 0.00 GB write, 18.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:01:12.138 INFO:journalctl@ceph.mon.a.smithi174.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T22:01:12.138 INFO:journalctl@ceph.mon.a.smithi174.stdout: 2024-04-22T22:01:12.138 INFO:journalctl@ceph.mon.a.smithi174.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T22:01:12.138 INFO:journalctl@ceph.mon.a.smithi174.stdout: 2024-04-22T22:01:12.138 INFO:journalctl@ceph.mon.a.smithi174.stdout: ** Compaction Stats [default] ** 2024-04-22T22:01:12.138 INFO:journalctl@ceph.mon.a.smithi174.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:01:12.138 INFO:journalctl@ceph.mon.a.smithi174.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:01:12.138 INFO:journalctl@ceph.mon.a.smithi174.stdout: L0 2/0 79.11 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.8 0.00 0.00 1 0.001 0 0 2024-04-22T22:01:12.138 INFO:journalctl@ceph.mon.a.smithi174.stdout: Sum 2/0 79.11 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 114.8 0.00 0.00 1 0.001 0 0 2024-04-22T22:01:12.138 INFO:journalctl@ceph.mon.a.smithi174.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-22T22:01:12.139 INFO:journalctl@ceph.mon.a.smithi174.stdout: 2024-04-22T22:01:12.139 INFO:journalctl@ceph.mon.a.smithi174.stdout: ** Compaction Stats [default] ** 2024-04-22T22:01:12.139 INFO:journalctl@ceph.mon.a.smithi174.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:01:12.139 INFO:journalctl@ceph.mon.a.smithi174.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:01:12.139 INFO:journalctl@ceph.mon.a.smithi174.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 114.8 0.00 0.00 1 0.001 0 0 2024-04-22T22:01:12.139 INFO:journalctl@ceph.mon.a.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:01:12.139 INFO:journalctl@ceph.mon.a.smithi174.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T22:01:12.139 INFO:journalctl@ceph.mon.a.smithi174.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T22:01:12.139 INFO:journalctl@ceph.mon.a.smithi174.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T22:01:12.139 INFO:journalctl@ceph.mon.a.smithi174.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T22:01:12.139 INFO:journalctl@ceph.mon.a.smithi174.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T22:01:12.140 INFO:journalctl@ceph.mon.a.smithi174.stdout: Cumulative compaction: 0.00 GB write, 18.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:01:12.140 INFO:journalctl@ceph.mon.a.smithi174.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:01:12.140 INFO:journalctl@ceph.mon.a.smithi174.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T22:01:12.140 INFO:journalctl@ceph.mon.a.smithi174.stdout: 2024-04-22T22:01:12.140 INFO:journalctl@ceph.mon.a.smithi174.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T22:01:12.140 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: starting mon.a rank 0 at public addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] at bind addrs [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:01:12.140 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: mon.a@-1(???) e1 preinit fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:01:12.140 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: mon.a@-1(???).mds e1 new map 2024-04-22T22:01:12.140 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: mon.a@-1(???).mds e1 print_map 2024-04-22T22:01:12.140 INFO:journalctl@ceph.mon.a.smithi174.stdout: e1 2024-04-22T22:01:12.140 INFO:journalctl@ceph.mon.a.smithi174.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T22:01:12.140 INFO:journalctl@ceph.mon.a.smithi174.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T22:01:12.140 INFO:journalctl@ceph.mon.a.smithi174.stdout: legacy client fscid: -1 2024-04-22T22:01:12.141 INFO:journalctl@ceph.mon.a.smithi174.stdout: 2024-04-22T22:01:12.141 INFO:journalctl@ceph.mon.a.smithi174.stdout: No filesystems configured 2024-04-22T22:01:12.141 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T22:01:12.141 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:01:12.141 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:01:12.141 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:01:12.141 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-22T22:01:12.141 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: expand_channel_meta expand map: {default=false} 2024-04-22T22:01:12.141 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: expand_channel_meta from 'false' to 'false' 2024-04-22T22:01:12.141 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: expand_channel_meta expanded map: {default=false} 2024-04-22T22:01:12.141 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: expand_channel_meta expand map: {default=info} 2024-04-22T22:01:12.141 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: expand_channel_meta from 'info' to 'info' 2024-04-22T22:01:12.141 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: expand_channel_meta expanded map: {default=info} 2024-04-22T22:01:12.142 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: expand_channel_meta expand map: {default=daemon} 2024-04-22T22:01:12.142 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-22T22:01:12.142 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: expand_channel_meta expanded map: {default=daemon} 2024-04-22T22:01:12.142 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: expand_channel_meta expand map: {default=debug} 2024-04-22T22:01:12.142 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: expand_channel_meta from 'debug' to 'debug' 2024-04-22T22:01:12.142 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:11 smithi174 ceph-mon[21825]: expand_channel_meta expanded map: {default=debug} 2024-04-22T22:01:12.142 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:12 smithi174 bash[21807]: d85a5efa7dc5763848bb269686567df898f06fd442d4ff43a9b6d16923597215 2024-04-22T22:01:12.142 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:12 smithi174 systemd[1]: Started Ceph mon.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:01:12.142 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:12 smithi174 ceph-mon[21825]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-22T22:01:12.142 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:12 smithi174 ceph-mon[21825]: monmap e1: 1 mons at {a=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0]} 2024-04-22T22:01:12.142 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:12 smithi174 ceph-mon[21825]: fsmap 2024-04-22T22:01:12.142 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:12 smithi174 ceph-mon[21825]: osdmap e1: 0 total, 0 up, 0 in 2024-04-22T22:01:12.143 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:12 smithi174 ceph-mon[21825]: mgrmap e1: no daemons active 2024-04-22T22:01:13.099 INFO:teuthology.orchestra.run.smithi174.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-22T22:01:13.099 INFO:teuthology.orchestra.run.smithi174.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:01:13.099 INFO:teuthology.orchestra.run.smithi174.stderr:Creating mgr... 2024-04-22T22:01:13.099 INFO:teuthology.orchestra.run.smithi174.stderr:Verifying port 9283 ... 2024-04-22T22:01:13.343 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: Failed to reset failed state of unit ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mgr.y.service: Unit ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mgr.y.service not loaded. 2024-04-22T22:01:13.350 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: Created symlink /etc/systemd/system/ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968.target.wants/ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@.service. 2024-04-22T22:01:13.706 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:13 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1472488351' entity='client.admin' 2024-04-22T22:01:14.032 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:13 smithi174 podman[22044]: 2024-04-22 22:01:13.958790672 +0000 UTC m=+0.338071732 container start 739b59b845dcb92dad8da069947f1db01e078d686ccc4e38685d4ac5a462a7b0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, distribution-scope=public) 2024-04-22T22:01:14.047 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: disabled 2024-04-22T22:01:14.056 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: inactive 2024-04-22T22:01:14.056 INFO:teuthology.orchestra.run.smithi174.stderr:firewalld.service is not enabled 2024-04-22T22:01:14.056 INFO:teuthology.orchestra.run.smithi174.stderr:Not possible to enable service . firewalld.service is not available 2024-04-22T22:01:14.066 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: disabled 2024-04-22T22:01:14.074 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: inactive 2024-04-22T22:01:14.075 INFO:teuthology.orchestra.run.smithi174.stderr:firewalld.service is not enabled 2024-04-22T22:01:14.075 INFO:teuthology.orchestra.run.smithi174.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-22T22:01:14.075 INFO:teuthology.orchestra.run.smithi174.stderr:Waiting for mgr to start... 2024-04-22T22:01:14.075 INFO:teuthology.orchestra.run.smithi174.stderr:Waiting for mgr... 2024-04-22T22:01:14.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:14 smithi174 bash[22044]: 739b59b845dcb92dad8da069947f1db01e078d686ccc4e38685d4ac5a462a7b0 2024-04-22T22:01:14.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:14 smithi174 systemd[1]: Started Ceph mgr.y for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:01:14.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:14 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:14.108+0000 7f258e0a6000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T22:01:14.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:14 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:14.243+0000 7f258e0a6000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T22:01:14.566 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:14 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:14.341+0000 7f258e0a6000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T22:01:14.606 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 2024-04-22T22:01:14.606 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: { 2024-04-22T22:01:14.606 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "fsid": "b7cbfdbc-00f3-11ef-bc93-c7b262605968", 2024-04-22T22:01:14.606 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "health": { 2024-04-22T22:01:14.606 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-22T22:01:14.606 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "checks": {}, 2024-04-22T22:01:14.606 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "mutes": [] 2024-04-22T22:01:14.607 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:14.607 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-22T22:01:14.607 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "quorum": [ 2024-04-22T22:01:14.607 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 0 2024-04-22T22:01:14.607 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: ], 2024-04-22T22:01:14.607 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-22T22:01:14.607 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "a" 2024-04-22T22:01:14.607 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: ], 2024-04-22T22:01:14.607 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-04-22T22:01:14.607 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "monmap": { 2024-04-22T22:01:14.607 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "osdmap": { 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "pgmap": { 2024-04-22T22:01:14.608 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "fsmap": { 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "available": false, 2024-04-22T22:01:14.609 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "modules": [ 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "iostat", 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "nfs", 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "restful" 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: ], 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "servicemap": { 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "modified": "2024-04-22T22:01:07.496695+0000", 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-22T22:01:14.610 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: } 2024-04-22T22:01:14.913 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:14 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/683211473' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T22:01:14.913 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:14 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:14.671+0000 7f258e0a6000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T22:01:14.914 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:14 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:14.752+0000 7f258e0a6000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T22:01:15.216 INFO:teuthology.orchestra.run.smithi174.stderr:mgr not available, waiting (1/15)... 2024-04-22T22:01:16.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:15 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:15.880+0000 7f258e0a6000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T22:01:16.316 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:16 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:16.041+0000 7f258e0a6000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T22:01:16.316 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:16 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:16.121+0000 7f258e0a6000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T22:01:16.816 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:16 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:16.430+0000 7f258e0a6000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T22:01:16.816 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:16 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:16.517+0000 7f258e0a6000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T22:01:16.816 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:16 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:16.608+0000 7f258e0a6000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T22:01:17.272 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:16 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:16.850+0000 7f258e0a6000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T22:01:17.566 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:17 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:17.295+0000 7f258e0a6000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T22:01:17.783 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 2024-04-22T22:01:17.783 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: { 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "fsid": "b7cbfdbc-00f3-11ef-bc93-c7b262605968", 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "health": { 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "checks": {}, 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "mutes": [] 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "quorum": [ 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 0 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: ], 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "a" 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: ], 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-04-22T22:01:17.784 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "monmap": { 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "osdmap": { 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:17.785 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "pgmap": { 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "fsmap": { 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:17.786 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "available": false, 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "modules": [ 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "iostat", 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "nfs", 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "restful" 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: ], 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "servicemap": { 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "modified": "2024-04-22T22:01:07.496695+0000", 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:17.787 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-22T22:01:17.788 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: } 2024-04-22T22:01:17.868 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:17 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:17.791+0000 7f258e0a6000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T22:01:17.868 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:17 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:17.891+0000 7f258e0a6000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T22:01:18.160 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:17 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2649447278' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T22:01:18.560 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:18 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:18.184+0000 7f258e0a6000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T22:01:18.561 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:18 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:18.270+0000 7f258e0a6000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T22:01:18.815 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:18 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:18.675+0000 7f258e0a6000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T22:01:18.816 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:18 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:18.805+0000 7f258e0a6000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T22:01:19.174 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:18 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:18.894+0000 7f258e0a6000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T22:01:19.174 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:19 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:19.007+0000 7f258e0a6000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T22:01:19.174 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:19 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:19.093+0000 7f258e0a6000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T22:01:19.177 INFO:teuthology.orchestra.run.smithi174.stderr:mgr not available, waiting (2/15)... 2024-04-22T22:01:19.973 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:19 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:19.634+0000 7f258e0a6000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T22:01:20.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:19 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:19.995+0000 7f258e0a6000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T22:01:20.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:20 smithi174 ceph-mon[21825]: Activating manager daemon y 2024-04-22T22:01:20.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:20 smithi174 ceph-mon[21825]: mgrmap e2: y(active, starting, since 0.00277352s) 2024-04-22T22:01:20.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:20 smithi174 ceph-mon[21825]: from='mgr.14100 172.21.15.174:0/3736573274' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T22:01:20.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:20 smithi174 ceph-mon[21825]: from='mgr.14100 172.21.15.174:0/3736573274' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T22:01:20.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:20 smithi174 ceph-mon[21825]: from='mgr.14100 172.21.15.174:0/3736573274' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T22:01:20.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:20 smithi174 ceph-mon[21825]: from='mgr.14100 172.21.15.174:0/3736573274' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:01:20.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:20 smithi174 ceph-mon[21825]: from='mgr.14100 172.21.15.174:0/3736573274' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T22:01:20.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:20 smithi174 ceph-mon[21825]: Manager daemon y is now available 2024-04-22T22:01:20.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:20 smithi174 ceph-mon[21825]: from='mgr.14100 172.21.15.174:0/3736573274' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:01:20.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:20 smithi174 ceph-mon[21825]: from='mgr.14100 172.21.15.174:0/3736573274' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:01:20.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:20 smithi174 ceph-mon[21825]: from='mgr.14100 172.21.15.174:0/3736573274' entity='mgr.y' 2024-04-22T22:01:20.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:20 smithi174 ceph-mon[21825]: from='mgr.14100 172.21.15.174:0/3736573274' entity='mgr.y' 2024-04-22T22:01:20.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:20 smithi174 ceph-mon[21825]: from='mgr.14100 172.21.15.174:0/3736573274' entity='mgr.y' 2024-04-22T22:01:21.848 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 2024-04-22T22:01:21.848 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: { 2024-04-22T22:01:21.849 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "fsid": "b7cbfdbc-00f3-11ef-bc93-c7b262605968", 2024-04-22T22:01:21.849 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "health": { 2024-04-22T22:01:21.849 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-22T22:01:21.849 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "checks": {}, 2024-04-22T22:01:21.849 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "mutes": [] 2024-04-22T22:01:21.849 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:21.849 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-22T22:01:21.849 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "quorum": [ 2024-04-22T22:01:21.849 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 0 2024-04-22T22:01:21.849 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: ], 2024-04-22T22:01:21.849 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "a" 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: ], 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "quorum_age": 9, 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "monmap": { 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "osdmap": { 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-22T22:01:21.850 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-22T22:01:21.854 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-22T22:01:21.854 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-22T22:01:21.854 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:21.854 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "pgmap": { 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "fsmap": { 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:21.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "available": true, 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "modules": [ 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "iostat", 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "nfs", 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "restful" 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: ], 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "servicemap": { 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "modified": "2024-04-22T22:01:07.496695+0000", 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "services": {} 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: }, 2024-04-22T22:01:21.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-22T22:01:21.857 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: } 2024-04-22T22:01:22.199 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:22 smithi174 ceph-mon[21825]: mgrmap e3: y(active, since 1.00668s) 2024-04-22T22:01:22.199 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:22 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/4275596235' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-22T22:01:22.741 INFO:teuthology.orchestra.run.smithi174.stderr:mgr is available 2024-04-22T22:01:23.335 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 2024-04-22T22:01:23.336 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: [global] 2024-04-22T22:01:23.336 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: fsid = b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:01:23.336 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-22T22:01:23.336 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-22T22:01:23.336 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-22T22:01:23.336 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-22T22:01:23.336 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 2024-04-22T22:01:23.336 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: [mgr] 2024-04-22T22:01:23.336 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-22T22:01:23.336 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 2024-04-22T22:01:23.336 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: [osd] 2024-04-22T22:01:23.336 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-22T22:01:23.337 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-22T22:01:23.913 INFO:teuthology.orchestra.run.smithi174.stderr:Enabling cephadm module... 2024-04-22T22:01:24.102 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:23 smithi174 ceph-mon[21825]: mgrmap e4: y(active, since 2s) 2024-04-22T22:01:24.102 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:23 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3734783451' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-22T22:01:24.102 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:23 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3734783451' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-22T22:01:25.258 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:24 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3363148059' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-22T22:01:25.258 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:25 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ignoring --setuser ceph since I am not root 2024-04-22T22:01:25.258 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:25 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ignoring --setgroup ceph since I am not root 2024-04-22T22:01:25.258 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:25 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:25.153+0000 7f4bad2c8000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T22:01:25.258 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:25 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:25.281+0000 7f4bad2c8000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T22:01:25.550 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:25 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:25.378+0000 7f4bad2c8000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T22:01:25.804 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:25 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:25.690+0000 7f4bad2c8000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T22:01:25.805 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:25 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:25.773+0000 7f4bad2c8000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T22:01:26.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:25 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3363148059' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-22T22:01:26.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:25 smithi174 ceph-mon[21825]: mgrmap e5: y(active, since 4s) 2024-04-22T22:01:26.278 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: { 2024-04-22T22:01:26.278 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-22T22:01:26.278 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "available": true, 2024-04-22T22:01:26.278 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-22T22:01:26.278 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-22T22:01:26.278 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: } 2024-04-22T22:01:26.858 INFO:teuthology.orchestra.run.smithi174.stderr:Waiting for the mgr to restart... 2024-04-22T22:01:26.859 INFO:teuthology.orchestra.run.smithi174.stderr:Waiting for mgr epoch 5... 2024-04-22T22:01:27.142 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:26 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/461612714' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T22:01:27.143 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:26 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:26.912+0000 7f4bad2c8000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T22:01:27.143 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:27 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:27.081+0000 7f4bad2c8000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T22:01:27.143 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:27 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:27.166+0000 7f4bad2c8000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T22:01:27.816 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:27 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:27.482+0000 7f4bad2c8000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T22:01:27.816 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:27 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:27.572+0000 7f4bad2c8000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T22:01:27.816 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:27 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:27.664+0000 7f4bad2c8000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T22:01:28.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:27 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:27.906+0000 7f4bad2c8000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T22:01:28.799 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:28 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:28.340+0000 7f4bad2c8000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T22:01:28.799 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:28 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:28.822+0000 7f4bad2c8000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T22:01:29.065 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:28 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:28.913+0000 7f4bad2c8000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T22:01:29.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:29 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:29.201+0000 7f4bad2c8000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T22:01:29.566 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:29 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:29.288+0000 7f4bad2c8000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T22:01:30.006 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:29 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:29.695+0000 7f4bad2c8000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T22:01:30.006 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:29 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:29.825+0000 7f4bad2c8000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T22:01:30.006 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:29 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:29.915+0000 7f4bad2c8000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T22:01:30.006 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:30 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:30.028+0000 7f4bad2c8000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T22:01:30.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:30 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:30.116+0000 7f4bad2c8000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T22:01:31.001 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:30 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:30.656+0000 7f4bad2c8000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T22:01:31.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:31 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:31.024+0000 7f4bad2c8000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T22:01:31.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:31 smithi174 ceph-mon[21825]: Active manager daemon y restarted 2024-04-22T22:01:31.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:31 smithi174 ceph-mon[21825]: Activating manager daemon y 2024-04-22T22:01:31.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:31 smithi174 ceph-mon[21825]: osdmap e2: 0 total, 0 up, 0 in 2024-04-22T22:01:32.069 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: { 2024-04-22T22:01:32.069 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-22T22:01:32.070 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "initialized": true 2024-04-22T22:01:32.070 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: } 2024-04-22T22:01:32.277 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: [22/Apr/2024:22:01:32] ENGINE Bus STARTING 2024-04-22T22:01:32.277 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: [22/Apr/2024:22:01:32] ENGINE Serving on https://172.21.15.174:7150 2024-04-22T22:01:32.277 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: mgrmap e6: y(active, starting, since 0.0529792s) 2024-04-22T22:01:32.277 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:01:32.277 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T22:01:32.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T22:01:32.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T22:01:32.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T22:01:32.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: Manager daemon y is now available 2024-04-22T22:01:32.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:32.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:32.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:32.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:01:32.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:01:32.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:01:32.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:01:32.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:01:32.553 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:32 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: [22/Apr/2024:22:01:32] ENGINE Bus STARTED 2024-04-22T22:01:32.731 INFO:teuthology.orchestra.run.smithi174.stderr:mgr epoch 5 is available 2024-04-22T22:01:32.731 INFO:teuthology.orchestra.run.smithi174.stderr:Setting orchestrator backend to cephadm... 2024-04-22T22:01:33.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:33 smithi174 ceph-mon[21825]: mgrmap e7: y(active, since 1.06018s) 2024-04-22T22:01:33.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:33 smithi174 ceph-mon[21825]: [22/Apr/2024:22:01:32] ENGINE Bus STARTING 2024-04-22T22:01:33.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:33 smithi174 ceph-mon[21825]: [22/Apr/2024:22:01:32] ENGINE Serving on https://172.21.15.174:7150 2024-04-22T22:01:33.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:33 smithi174 ceph-mon[21825]: [22/Apr/2024:22:01:32] ENGINE Bus STARTED 2024-04-22T22:01:33.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:33 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:33.107 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:33 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:01:34.510 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:34 smithi174 ceph-mon[21825]: mgrmap e8: y(active, since 2s) 2024-04-22T22:01:34.510 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:34 smithi174 ceph-mon[21825]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:01:34.510 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:34 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:34.510 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:34 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:01:34.510 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: value unchanged 2024-04-22T22:01:35.043 INFO:teuthology.orchestra.run.smithi174.stderr:Generating ssh key... 2024-04-22T22:01:35.493 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-mon[21825]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:01:36.163 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: Generating public/private rsa key pair. 2024-04-22T22:01:36.163 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: Your identification has been saved in /tmp/tmpzn_umuu9/key. 2024-04-22T22:01:36.163 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: Your public key has been saved in /tmp/tmpzn_umuu9/key.pub. 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: The key fingerprint is: 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: SHA256:MA9c3/BMer9ydPr0lP2tN9Gfy61p9u8J5qgCVTG7lGI ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: The key's randomart image is: 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: +---[RSA 3072]----+ 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: | .oo . | 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: | . . o+B | 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: | =E.+o = | 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: | .*o .. . | 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: | . S. o o| 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: | . . =+| 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: | . .o+.B| 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: | . +o=BO| 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: | .... ++X&| 2024-04-22T22:01:36.164 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: +----[SHA256]-----+ 2024-04-22T22:01:36.996 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:36 smithi174 ceph-mon[21825]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:01:36.996 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:36 smithi174 ceph-mon[21825]: Generating ssh key... 2024-04-22T22:01:36.997 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:36 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:36.997 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:36 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:36.997 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1nuAZLV/ajPQtEUa0zHKeDdWtnVpZJu46a34Z9o8X84dzVxGlAS+XICj54nJCZpKakh5vNIIP34v32XnACIH9tStqijNUZ6yMQz15Qj84e4PS2lpToFAeLOEyBKvyfQdiuW/yeGnahdiX5PrjOna3lGyHca2pwtYPHCqhLWLh0dfbNkIsCKqKPSBERyHFebp46BJfMwCASDzbbJnMSHLrTKKzewiBzWjbBM4cCuwdAg3JDorqNCXuWEBpZwYZrPzH5NUQ6iHi+06ftN8oz7Z0DwdPQ7Uagrj+Jyl0hEHJw7fXfX2Ev44vAjDI/yqabPuAadPPmnJZ2ECRC+EVCp4oe3I/1gRf8OUWBUr3gVJveL5HlJYS++TNkDA0WmOf0jXUupi7WPvhQpZaE+tE6KsT4bibxemLp6BdF5GVr5lOEPmE60tKzq3VBzfpz4F7wDVLCYjNgSeUL60t3g2FFTRKqK+qETIlRCV8wn4PB0/ii7g5yKkrm5EI4KvOnzrPXoc= ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:01:37.580 INFO:teuthology.orchestra.run.smithi174.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-22T22:01:37.581 INFO:teuthology.orchestra.run.smithi174.stderr:Adding key to root@localhost authorized_keys... 2024-04-22T22:01:37.581 INFO:teuthology.orchestra.run.smithi174.stderr:Adding host smithi174... 2024-04-22T22:01:38.213 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:37 smithi174 ceph-mon[21825]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:01:39.060 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: Added host 'smithi174' with addr '172.21.15.174' 2024-04-22T22:01:39.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:39 smithi174 ceph-mon[21825]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi174", "addr": "172.21.15.174", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:01:39.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:39 smithi174 ceph-mon[21825]: Deploying cephadm binary to smithi174 2024-04-22T22:01:39.126 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:39 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:39.654 INFO:teuthology.orchestra.run.smithi174.stderr:Deploying unmanaged mon service... 2024-04-22T22:01:40.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:40 smithi174 ceph-mon[21825]: Added host smithi174 2024-04-22T22:01:40.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:40 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:01:40.343 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-22T22:01:41.011 INFO:teuthology.orchestra.run.smithi174.stderr:Deploying unmanaged mgr service... 2024-04-22T22:01:41.521 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:41 smithi174 ceph-mon[21825]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:01:41.522 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:41 smithi174 ceph-mon[21825]: Saving service mon spec with placement count:5 2024-04-22T22:01:41.522 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:41 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:41.833 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-22T22:01:43.168 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:42 smithi174 ceph-mon[21825]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:01:43.169 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:42 smithi174 ceph-mon[21825]: Saving service mgr spec with placement count:2 2024-04-22T22:01:43.169 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:42 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:44.465 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:44 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:44.465 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:44 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:44.465 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:44 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1073801726' entity='client.admin' 2024-04-22T22:01:46.120 INFO:teuthology.orchestra.run.smithi174.stderr:Enabling the dashboard module... 2024-04-22T22:01:46.747 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:46 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:46.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:46 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:46.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:46 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:01:46.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:46 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' 2024-04-22T22:01:46.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:46 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi174", "caps": []}]: dispatch 2024-04-22T22:01:46.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:46 smithi174 ceph-mon[21825]: from='mgr.14118 172.21.15.174:0/415906755' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi174", "caps": []}]': finished 2024-04-22T22:01:46.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:46 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2563380119' entity='client.admin' 2024-04-22T22:01:47.741 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:47 smithi174 ceph-mon[21825]: Deploying daemon agent.smithi174 on smithi174 2024-04-22T22:01:47.741 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:47 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1753835172' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-22T22:01:47.741 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:47 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ignoring --setuser ceph since I am not root 2024-04-22T22:01:47.741 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:47 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ignoring --setgroup ceph since I am not root 2024-04-22T22:01:47.741 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:47 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:47.633+0000 7ff33c102000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T22:01:47.741 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:47 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:47.763+0000 7ff33c102000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T22:01:48.065 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:47 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:47.863+0000 7ff33c102000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T22:01:48.368 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:48 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:48.175+0000 7ff33c102000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T22:01:48.368 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:48 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:48.259+0000 7ff33c102000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T22:01:48.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:48 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1753835172' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-22T22:01:48.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:48 smithi174 ceph-mon[21825]: mgrmap e9: y(active, since 16s) 2024-04-22T22:01:48.855 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: { 2024-04-22T22:01:48.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-22T22:01:48.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "available": true, 2024-04-22T22:01:48.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-22T22:01:48.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-22T22:01:48.856 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: } 2024-04-22T22:01:49.530 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:49 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3667504363' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-22T22:01:49.531 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:49 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:49.407+0000 7ff33c102000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T22:01:49.533 INFO:teuthology.orchestra.run.smithi174.stderr:Waiting for the mgr to restart... 2024-04-22T22:01:49.533 INFO:teuthology.orchestra.run.smithi174.stderr:Waiting for mgr epoch 9... 2024-04-22T22:01:49.802 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:49 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:49.577+0000 7ff33c102000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T22:01:49.802 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:49 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:49.664+0000 7ff33c102000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T22:01:50.059 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:49 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:49.988+0000 7ff33c102000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T22:01:50.059 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:50 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:50.082+0000 7ff33c102000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T22:01:50.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:50 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:50.179+0000 7ff33c102000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T22:01:50.815 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:50 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:50.432+0000 7ff33c102000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T22:01:51.244 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:50 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:50.887+0000 7ff33c102000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T22:01:51.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:51 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:51.384+0000 7ff33c102000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T22:01:51.566 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:51 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:51.478+0000 7ff33c102000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T22:01:52.065 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:51 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:51.771+0000 7ff33c102000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T22:01:52.066 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:51 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:51.861+0000 7ff33c102000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T22:01:52.566 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:52 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:52.273+0000 7ff33c102000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T22:01:52.566 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:52 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:52.404+0000 7ff33c102000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T22:01:52.566 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:52 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:52.496+0000 7ff33c102000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T22:01:53.065 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:52 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:52.610+0000 7ff33c102000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T22:01:53.066 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:52 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:52.697+0000 7ff33c102000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T22:01:53.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:53 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:53.237+0000 7ff33c102000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T22:01:53.824 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:53 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:01:53.597+0000 7ff33c102000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T22:01:53.824 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:53 smithi174 ceph-mon[21825]: Active manager daemon y restarted 2024-04-22T22:01:53.824 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:53 smithi174 ceph-mon[21825]: Activating manager daemon y 2024-04-22T22:01:53.824 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:53 smithi174 ceph-mon[21825]: osdmap e3: 0 total, 0 up, 0 in 2024-04-22T22:01:54.647 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: { 2024-04-22T22:01:54.647 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-22T22:01:54.647 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: "initialized": true 2024-04-22T22:01:54.647 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: } 2024-04-22T22:01:54.924 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: [22/Apr/2024:22:01:54] ENGINE Bus STARTING 2024-04-22T22:01:54.924 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: [22/Apr/2024:22:01:54] ENGINE Serving on https://172.21.15.174:7150 2024-04-22T22:01:54.924 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: [22/Apr/2024:22:01:54] ENGINE Bus STARTED 2024-04-22T22:01:54.924 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-mon[21825]: mgrmap e10: y(active, starting, since 0.0550827s) 2024-04-22T22:01:54.925 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:01:54.925 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T22:01:54.925 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T22:01:54.925 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T22:01:54.925 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T22:01:54.925 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-mon[21825]: Manager daemon y is now available 2024-04-22T22:01:54.925 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:01:54.925 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:01:54.925 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:01:54.925 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:01:54.925 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:01:55.318 INFO:teuthology.orchestra.run.smithi174.stderr:mgr epoch 9 is available 2024-04-22T22:01:55.318 INFO:teuthology.orchestra.run.smithi174.stderr:Generating a dashboard self-signed certificate... 2024-04-22T22:01:55.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:55 smithi174 ceph-mon[21825]: [22/Apr/2024:22:01:54] ENGINE Bus STARTING 2024-04-22T22:01:55.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:55 smithi174 ceph-mon[21825]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-22T22:01:55.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:55 smithi174 ceph-mon[21825]: mgrmap e11: y(active, since 1.0653s) 2024-04-22T22:01:55.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:55 smithi174 ceph-mon[21825]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-22T22:01:55.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:55 smithi174 ceph-mon[21825]: [22/Apr/2024:22:01:54] ENGINE Serving on https://172.21.15.174:7150 2024-04-22T22:01:55.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:55 smithi174 ceph-mon[21825]: [22/Apr/2024:22:01:54] ENGINE Bus STARTED 2024-04-22T22:01:55.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:01:55.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:01:55.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:01:55.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:01:55.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi174", "caps": []}]: dispatch 2024-04-22T22:01:55.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:55 smithi174 ceph-mon[21825]: Deploying daemon agent.smithi174 on smithi174 2024-04-22T22:01:56.001 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-22T22:01:56.625 INFO:teuthology.orchestra.run.smithi174.stderr:Creating initial admin user... 2024-04-22T22:01:57.050 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:56 smithi174 ceph-mon[21825]: mgrmap e12: y(active, since 2s) 2024-04-22T22:01:57.050 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:56 smithi174 ceph-mon[21825]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:01:57.050 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:56 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:01:57.050 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:56 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:01:57.569 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$Q9aWBBN8fo9.KCPGVZs.2.KGI7yxNihSk2Q4BJDbud3jNTRcg9LtC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713823317, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-22T22:01:58.204 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:57 smithi174 ceph-mon[21825]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:01:58.204 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:57 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:01:58.213 INFO:teuthology.orchestra.run.smithi174.stderr:Fetching dashboard port number... 2024-04-22T22:01:58.898 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: 8443 2024-04-22T22:01:59.201 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:01:58 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3196003187' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-22T22:01:59.514 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: disabled 2024-04-22T22:01:59.521 INFO:teuthology.orchestra.run.smithi174.stderr:systemctl: inactive 2024-04-22T22:01:59.521 INFO:teuthology.orchestra.run.smithi174.stderr:firewalld.service is not enabled 2024-04-22T22:01:59.521 INFO:teuthology.orchestra.run.smithi174.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-22T22:01:59.523 INFO:teuthology.orchestra.run.smithi174.stderr:Ceph Dashboard is now available at: 2024-04-22T22:01:59.523 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-22T22:01:59.523 INFO:teuthology.orchestra.run.smithi174.stderr: URL: https://smithi174:8443/ 2024-04-22T22:01:59.523 INFO:teuthology.orchestra.run.smithi174.stderr: User: admin 2024-04-22T22:01:59.523 INFO:teuthology.orchestra.run.smithi174.stderr: Password: 4ya6y249sl 2024-04-22T22:01:59.524 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-22T22:01:59.524 INFO:teuthology.orchestra.run.smithi174.stderr:Enabling autotune for osd_memory_target 2024-04-22T22:02:01.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:00 smithi174 ceph-mon[21825]: mgrmap e13: y(active, since 6s) 2024-04-22T22:02:01.534 INFO:teuthology.orchestra.run.smithi174.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-22T22:02:02.210 INFO:teuthology.orchestra.run.smithi174.stderr:You can access the Ceph CLI with: 2024-04-22T22:02:02.210 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-22T22:02:02.210 INFO:teuthology.orchestra.run.smithi174.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:02:02.210 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-22T22:02:02.210 INFO:teuthology.orchestra.run.smithi174.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-22T22:02:02.210 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-22T22:02:02.210 INFO:teuthology.orchestra.run.smithi174.stderr: ceph telemetry on 2024-04-22T22:02:02.211 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-22T22:02:02.211 INFO:teuthology.orchestra.run.smithi174.stderr:For more information see: 2024-04-22T22:02:02.211 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-22T22:02:02.211 INFO:teuthology.orchestra.run.smithi174.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-22T22:02:02.211 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-22T22:02:02.211 INFO:teuthology.orchestra.run.smithi174.stderr:Bootstrap complete. 2024-04-22T22:02:02.391 INFO:tasks.cephadm:Fetching config... 2024-04-22T22:02:02.391 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-22T22:02:02.391 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-22T22:02:02.406 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-22T22:02:02.406 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-22T22:02:02.406 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-22T22:02:02.463 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-22T22:02:02.465 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-22T22:02:02.465 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-22T22:02:02.579 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-22T22:02:02.579 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-22T22:02:02.579 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-22T22:02:02.636 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-22T22:02:02.637 DEBUG:teuthology.orchestra.run.smithi174:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1nuAZLV/ajPQtEUa0zHKeDdWtnVpZJu46a34Z9o8X84dzVxGlAS+XICj54nJCZpKakh5vNIIP34v32XnACIH9tStqijNUZ6yMQz15Qj84e4PS2lpToFAeLOEyBKvyfQdiuW/yeGnahdiX5PrjOna3lGyHca2pwtYPHCqhLWLh0dfbNkIsCKqKPSBERyHFebp46BJfMwCASDzbbJnMSHLrTKKzewiBzWjbBM4cCuwdAg3JDorqNCXuWEBpZwYZrPzH5NUQ6iHi+06ftN8oz7Z0DwdPQ7Uagrj+Jyl0hEHJw7fXfX2Ev44vAjDI/yqabPuAadPPmnJZ2ECRC+EVCp4oe3I/1gRf8OUWBUr3gVJveL5HlJYS++TNkDA0WmOf0jXUupi7WPvhQpZaE+tE6KsT4bibxemLp6BdF5GVr5lOEPmE60tKzq3VBzfpz4F7wDVLCYjNgSeUL60t3g2FFTRKqK+qETIlRCV8wn4PB0/ii7g5yKkrm5EI4KvOnzrPXoc= ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T22:02:02.756 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:02 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2128475956' entity='client.admin' 2024-04-22T22:02:02.825 INFO:teuthology.orchestra.run.smithi174.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1nuAZLV/ajPQtEUa0zHKeDdWtnVpZJu46a34Z9o8X84dzVxGlAS+XICj54nJCZpKakh5vNIIP34v32XnACIH9tStqijNUZ6yMQz15Qj84e4PS2lpToFAeLOEyBKvyfQdiuW/yeGnahdiX5PrjOna3lGyHca2pwtYPHCqhLWLh0dfbNkIsCKqKPSBERyHFebp46BJfMwCASDzbbJnMSHLrTKKzewiBzWjbBM4cCuwdAg3JDorqNCXuWEBpZwYZrPzH5NUQ6iHi+06ftN8oz7Z0DwdPQ7Uagrj+Jyl0hEHJw7fXfX2Ev44vAjDI/yqabPuAadPPmnJZ2ECRC+EVCp4oe3I/1gRf8OUWBUr3gVJveL5HlJYS++TNkDA0WmOf0jXUupi7WPvhQpZaE+tE6KsT4bibxemLp6BdF5GVr5lOEPmE60tKzq3VBzfpz4F7wDVLCYjNgSeUL60t3g2FFTRKqK+qETIlRCV8wn4PB0/ii7g5yKkrm5EI4KvOnzrPXoc= ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:02:02.985 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1nuAZLV/ajPQtEUa0zHKeDdWtnVpZJu46a34Z9o8X84dzVxGlAS+XICj54nJCZpKakh5vNIIP34v32XnACIH9tStqijNUZ6yMQz15Qj84e4PS2lpToFAeLOEyBKvyfQdiuW/yeGnahdiX5PrjOna3lGyHca2pwtYPHCqhLWLh0dfbNkIsCKqKPSBERyHFebp46BJfMwCASDzbbJnMSHLrTKKzewiBzWjbBM4cCuwdAg3JDorqNCXuWEBpZwYZrPzH5NUQ6iHi+06ftN8oz7Z0DwdPQ7Uagrj+Jyl0hEHJw7fXfX2Ev44vAjDI/yqabPuAadPPmnJZ2ECRC+EVCp4oe3I/1gRf8OUWBUr3gVJveL5HlJYS++TNkDA0WmOf0jXUupi7WPvhQpZaE+tE6KsT4bibxemLp6BdF5GVr5lOEPmE60tKzq3VBzfpz4F7wDVLCYjNgSeUL60t3g2FFTRKqK+qETIlRCV8wn4PB0/ii7g5yKkrm5EI4KvOnzrPXoc= ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-22T22:02:03.026 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1nuAZLV/ajPQtEUa0zHKeDdWtnVpZJu46a34Z9o8X84dzVxGlAS+XICj54nJCZpKakh5vNIIP34v32XnACIH9tStqijNUZ6yMQz15Qj84e4PS2lpToFAeLOEyBKvyfQdiuW/yeGnahdiX5PrjOna3lGyHca2pwtYPHCqhLWLh0dfbNkIsCKqKPSBERyHFebp46BJfMwCASDzbbJnMSHLrTKKzewiBzWjbBM4cCuwdAg3JDorqNCXuWEBpZwYZrPzH5NUQ6iHi+06ftN8oz7Z0DwdPQ7Uagrj+Jyl0hEHJw7fXfX2Ev44vAjDI/yqabPuAadPPmnJZ2ECRC+EVCp4oe3I/1gRf8OUWBUr3gVJveL5HlJYS++TNkDA0WmOf0jXUupi7WPvhQpZaE+tE6KsT4bibxemLp6BdF5GVr5lOEPmE60tKzq3VBzfpz4F7wDVLCYjNgSeUL60t3g2FFTRKqK+qETIlRCV8wn4PB0/ii7g5yKkrm5EI4KvOnzrPXoc= ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:02:03.038 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-22T22:02:03.620 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:03 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:02:03] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:04.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:04.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:04.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:04.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:04.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:04.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:04.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:04 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:06.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:06 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:02:06] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:07.406 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:07 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:08.418 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:08 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:02:08] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:08.442 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-22T22:02:08.442 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-22T22:02:09.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:08 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3147101517' entity='client.admin' 2024-04-22T22:02:09.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:08 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:09.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:08 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:09.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:08 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:09.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:08 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:11.032 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2024-04-22T22:02:11.032 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-22T22:02:11.032 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2024-04-22T22:02:11.050 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-22T22:02:11.050 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:02:11.107 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2024-04-22T22:02:11.108 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch host add smithi192 2024-04-22T22:02:11.629 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:11 smithi174 ceph-mon[21825]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:02:11.630 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:11.630 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:11.630 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:11.630 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:11.630 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:11 smithi174 ceph-mon[21825]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-22T22:02:11.630 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:11.630 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:12.594 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:12 smithi174 ceph-mon[21825]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:02:13.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:13 smithi174 ceph-mon[21825]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:02:13.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:13 smithi174 ceph-mon[21825]: Deploying cephadm binary to smithi192 2024-04-22T22:02:13.847 INFO:teuthology.orchestra.run.smithi174.stdout:Added host 'smithi192' with addr '172.21.15.192' 2024-04-22T22:02:14.448 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-22T22:02:15.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:14 smithi174 ceph-mon[21825]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:15.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:15.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:14 smithi174 ceph-mon[21825]: Added host smithi192 2024-04-22T22:02:15.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:15.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:15.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:15.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:15.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:15.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:15.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:02:15.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:15.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.agent.smithi192", "caps": []}]: dispatch 2024-04-22T22:02:15.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.agent.smithi192", "caps": []}]': finished 2024-04-22T22:02:16.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-mon[21825]: Deploying daemon agent.smithi192 on smithi192 2024-04-22T22:02:16.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:16.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:16.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:16.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:16.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-mon[21825]: Updating smithi192:/etc/ceph/ceph.conf 2024-04-22T22:02:16.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-mon[21825]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:02:16.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:16.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:16.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:16.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:16.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:16.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:16.387 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T22:02:16.387 INFO:teuthology.orchestra.run.smithi174.stdout:[{"addr": "172.21.15.174", "hostname": "smithi174", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}] 2024-04-22T22:02:16.687 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:16 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:02:16] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:16.962 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-22T22:02:16.962 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-22T22:02:17.265 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:17 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:02:17] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:17.557 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:17 smithi174 ceph-mon[21825]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:17.557 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:17 smithi174 ceph-mon[21825]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:02:17.558 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:17 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:17.558 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:17 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:17.558 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:17 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:18.417 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:18 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:02:18] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:19.684 INFO:teuthology.orchestra.run.smithi174.stderr:adjusted tunables profile to default 2024-04-22T22:02:19.685 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:19 smithi174 ceph-mon[21825]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:19.685 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:19 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:19.685 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:19 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:19.685 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:19 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:19.685 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:19 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2888699331' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-22T22:02:20.287 INFO:tasks.cephadm:Adding mon.a on smithi174 2024-04-22T22:02:20.287 INFO:tasks.cephadm:Adding mon.c on smithi174 2024-04-22T22:02:20.287 INFO:tasks.cephadm:Adding mon.b on smithi192 2024-04-22T22:02:20.288 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi174:172.21.15.174=a;smithi174:[v2:172.21.15.174:3301,v1:172.21.15.174:6790]=c;smithi192:172.21.15.192=b' 2024-04-22T22:02:21.067 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:20 smithi174 ceph-mon[21825]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:21.067 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:20 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2888699331' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-22T22:02:21.067 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:20 smithi174 ceph-mon[21825]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T22:02:22.052 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mon update... 2024-04-22T22:02:22.597 DEBUG:teuthology.orchestra.run.smithi174:mon.c> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mon.c.service 2024-04-22T22:02:22.600 DEBUG:teuthology.orchestra.run.smithi192:mon.b> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mon.b.service 2024-04-22T22:02:22.602 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T22:02:22.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:02:23.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:23 smithi174 ceph-mon[21825]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:23.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:23 smithi174 ceph-mon[21825]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi174:172.21.15.174=a;smithi174:[v2:172.21.15.174:3301,v1:172.21.15.174:6790]=c;smithi192:172.21.15.192=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:02:23.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:23 smithi174 ceph-mon[21825]: Saving service mon spec with placement smithi174:172.21.15.174=a;smithi174:[v2:172.21.15.174:3301,v1:172.21.15.174:6790]=c;smithi192:172.21.15.192=b;count:3 2024-04-22T22:02:23.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:23.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:23.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:23.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:23.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:23.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:23.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:23.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:23 smithi174 ceph-mon[21825]: Deploying daemon mon.c on smithi174 2024-04-22T22:02:24.476 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-22T22:02:24.477 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"b7cbfdbc-00f3-11ef-bc93-c7b262605968","modified":"2024-04-22T22:01:04.325314Z","created":"2024-04-22T22:01:04.325314Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-22T22:02:24.479 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-04-22T22:02:25.242 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:25 smithi174 ceph-mon[21825]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:25.242 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:25 smithi174 ceph-mon[21825]: from='client.? 172.21.15.192:0/1446062799' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:02:25.802 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:25 smithi174 podman[26676]: 2024-04-22 22:02:25.610411184 +0000 UTC m=+0.018954228 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:02:26.029 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T22:02:26.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:02:26.065 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:25 smithi174 podman[26676]: 2024-04-22 22:02:25.984574272 +0000 UTC m=+0.393117309 container init 260b4acfe3f883206d0cf7c6ef164c45b366648e664b1a8af0a77b7795514fe3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-c, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., distribution-scope=public, name=centos-stream, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, release=754, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.buildah.version=1.19.8, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream) 2024-04-22T22:02:26.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:25 smithi174 podman[26676]: 2024-04-22 22:02:25.987767269 +0000 UTC m=+0.396310321 container start 260b4acfe3f883206d0cf7c6ef164c45b366648e664b1a8af0a77b7795514fe3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-c, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, version=8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T22:02:26.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T22:02:26.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-22T22:02:26.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: pidfile_write: ignore empty --pid-file 2024-04-22T22:02:26.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: load: jerasure load: lrc 2024-04-22T22:02:26.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: RocksDB version: 6.15.5 2024-04-22T22:02:26.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-22T22:02:26.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Compile date Apr 18 2022 2024-04-22T22:02:26.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: DB SUMMARY 2024-04-22T22:02:26.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: DB Session ID: 1571YXYPWIY26PAMEKLX 2024-04-22T22:02:26.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: CURRENT file: CURRENT 2024-04-22T22:02:26.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T22:02:26.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-22T22:02:26.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-22T22:02:26.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-22T22:02:26.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.error_if_exists: 0 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.create_if_missing: 0 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.paranoid_checks: 1 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.env: 0x5639e16c6860 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.fs: Posix File System 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.info_log: 0x5639e28c5dc0 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.statistics: (nil) 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.use_fsync: 0 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_log_file_size: 0 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T22:02:26.068 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.allow_fallocate: 1 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.use_direct_reads: 0 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.db_log_dir: 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T22:02:26.069 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.write_buffer_manager: 0x5639e29b6270 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.unordered_write: 0 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T22:02:26.070 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T22:02:26.071 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T22:02:26.071 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.row_cache: None 2024-04-22T22:02:26.071 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.wal_filter: None 2024-04-22T22:02:26.071 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T22:02:26.071 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T22:02:26.071 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.preserve_deletes: 0 2024-04-22T22:02:26.071 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.two_write_queues: 0 2024-04-22T22:02:26.071 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T22:02:26.071 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.atomic_flush: 0 2024-04-22T22:02:26.071 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T22:02:26.071 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.log_readahead_size: 0 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_background_jobs: 2 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_background_compactions: -1 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_subcompactions: 1 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T22:02:26.072 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_open_files: -1 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_background_flushes: -1 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Compression algorithms supported: 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: kZSTD supported: 0 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: kXpressCompression supported: 0 2024-04-22T22:02:26.073 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: kLZ4Compression supported: 1 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: kBZip2Compression supported: 0 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: kZlibCompression supported: 1 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: kSnappyCompression supported: 1 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.merge_operator: 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compaction_filter: None 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T22:02:26.074 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5639e2891d00) 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: cache_index_and_filter_blocks: 1 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: pin_top_level_index_and_filter: 1 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: index_type: 0 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: data_block_index_type: 0 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: index_shortening: 1 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: hash_index_allow_collision: 1 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: checksum: 1 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: no_block_cache: 0 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_cache: 0x5639e28fc170 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_cache_name: BinnedLRUCache 2024-04-22T22:02:26.075 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_cache_options: 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: capacity : 536870912 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: num_shard_bits : 4 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: strict_capacity_limit : 0 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: high_pri_pool_ratio: 0.000 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_cache_compressed: (nil) 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: persistent_cache: (nil) 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_size: 4096 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_size_deviation: 10 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_restart_interval: 16 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: index_block_restart_interval: 1 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: metadata_block_size: 4096 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: partition_filters: 0 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: use_delta_encoding: 1 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: whole_key_filtering: 1 2024-04-22T22:02:26.076 INFO:journalctl@ceph.mon.c.smithi174.stdout: verify_compression: 0 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout: read_amp_bytes_per_bit: 0 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout: format_version: 4 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout: enable_index_compression: 1 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout: block_align: 0 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compression: NoCompression 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.num_levels: 7 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T22:02:26.077 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T22:02:26.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T22:02:26.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T22:02:26.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T22:02:26.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T22:02:26.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T22:02:26.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T22:02:26.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T22:02:26.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T22:02:26.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T22:02:26.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T22:02:26.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T22:02:26.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T22:02:26.078 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T22:02:26.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T22:02:26.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T22:02:26.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T22:02:26.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T22:02:26.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T22:02:26.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T22:02:26.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T22:02:26.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T22:02:26.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T22:02:26.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T22:02:26.080 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T22:02:26.081 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T22:02:26.081 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T22:02:26.081 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T22:02:26.081 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T22:02:26.081 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T22:02:26.081 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T22:02:26.081 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.arena_block_size: 4194304 2024-04-22T22:02:26.082 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T22:02:26.082 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.table_properties_collectors: 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.inplace_update_support: 0 2024-04-22T22:02:26.083 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.bloom_locality: 0 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.max_successive_merges: 0 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.ttl: 2592000 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.enable_blob_files: false 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.min_blob_size: 0 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T22:02:26.084 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713823346013666, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713823346013971, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713823346, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "73460bca-9e59-4e82-8c84-c51ebf3e591f", "db_session_id": "1571YXYPWIY26PAMEKLX"}} 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713823346014306, "job": 1, "event": "recovery_finished"} 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5639e28e2a80 2024-04-22T22:02:26.085 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: DB pointer 0x5639e28f2000 2024-04-22T22:02:26.086 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-22T22:02:26.086 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-22T22:02:26.086 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** DB Stats ** 2024-04-22T22:02:26.086 INFO:journalctl@ceph.mon.c.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:02:26.086 INFO:journalctl@ceph.mon.c.smithi174.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T22:02:26.086 INFO:journalctl@ceph.mon.c.smithi174.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T22:02:26.086 INFO:journalctl@ceph.mon.c.smithi174.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T22:02:26.087 INFO:journalctl@ceph.mon.c.smithi174.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T22:02:26.087 INFO:journalctl@ceph.mon.c.smithi174.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-22T22:02:26.087 INFO:journalctl@ceph.mon.c.smithi174.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T22:02:26.087 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-04-22T22:02:26.087 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** Compaction Stats [default] ** 2024-04-22T22:02:26.087 INFO:journalctl@ceph.mon.c.smithi174.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:02:26.087 INFO:journalctl@ceph.mon.c.smithi174.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:02:26.087 INFO:journalctl@ceph.mon.c.smithi174.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:26.087 INFO:journalctl@ceph.mon.c.smithi174.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:26.087 INFO:journalctl@ceph.mon.c.smithi174.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:26.088 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-04-22T22:02:26.088 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** Compaction Stats [default] ** 2024-04-22T22:02:26.088 INFO:journalctl@ceph.mon.c.smithi174.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:02:26.088 INFO:journalctl@ceph.mon.c.smithi174.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:02:26.088 INFO:journalctl@ceph.mon.c.smithi174.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:26.088 INFO:journalctl@ceph.mon.c.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:02:26.088 INFO:journalctl@ceph.mon.c.smithi174.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T22:02:26.088 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T22:02:26.088 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T22:02:26.088 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T22:02:26.088 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** Compaction Stats [default] ** 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** Compaction Stats [default] ** 2024-04-22T22:02:26.089 INFO:journalctl@ceph.mon.c.smithi174.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:02:26.090 INFO:journalctl@ceph.mon.c.smithi174.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:02:26.090 INFO:journalctl@ceph.mon.c.smithi174.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:26.090 INFO:journalctl@ceph.mon.c.smithi174.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:02:26.090 INFO:journalctl@ceph.mon.c.smithi174.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T22:02:26.090 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T22:02:26.090 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T22:02:26.090 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T22:02:26.091 INFO:journalctl@ceph.mon.c.smithi174.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T22:02:26.091 INFO:journalctl@ceph.mon.c.smithi174.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:02:26.091 INFO:journalctl@ceph.mon.c.smithi174.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:02:26.091 INFO:journalctl@ceph.mon.c.smithi174.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T22:02:26.091 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-04-22T22:02:26.091 INFO:journalctl@ceph.mon.c.smithi174.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T22:02:26.091 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-22T22:02:26.091 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: using public_addrv [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: starting mon.c rank -1 at public addrs [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] at bind addrs [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(???) e0 preinit fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing).mds e1 new map 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout: e1 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout: legacy client fscid: -1 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout: 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout: No filesystems configured 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T22:02:26.092 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi174:172.21.15.174=a;smithi174:[v2:172.21.15.174:3301,v1:172.21.15.174:6790]=c;smithi192:172.21.15.192=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: Saving service mon spec with placement smithi174:172.21.15.174=a;smithi174:[v2:172.21.15.174:3301,v1:172.21.15.174:6790]=c;smithi192:172.21.15.192=b;count:3 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:26.093 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:26.094 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:26.094 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: Deploying daemon mon.c on smithi174 2024-04-22T22:02:26.094 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:26.094 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: from='client.? 172.21.15.192:0/1446062799' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:02:26.094 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-04-22T22:02:26.094 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 ceph-mon[26730]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-22T22:02:26.094 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 bash[26676]: 260b4acfe3f883206d0cf7c6ef164c45b366648e664b1a8af0a77b7795514fe3 2024-04-22T22:02:26.094 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:26 smithi174 systemd[1]: Started Ceph mon.c for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:02:31.316 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:02:31] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:31.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:31.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:02:31.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: mon.a calling monitor election 2024-04-22T22:02:31.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:31.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:31.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:31.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:31.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: mon.c calling monitor election 2024-04-22T22:02:31.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:31.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:31.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:31.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:31.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T22:02:31.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: monmap e2: 2 mons at {a=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0]} 2024-04-22T22:02:31.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: fsmap 2024-04-22T22:02:31.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T22:02:31.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: mgrmap e13: y(active, since 37s) 2024-04-22T22:02:31.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: overall HEALTH_OK 2024-04-22T22:02:31.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:31.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:31.319 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:31.319 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:31.319 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:31.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:31.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:02:31.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: mon.a calling monitor election 2024-04-22T22:02:31.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:31.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:31.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:31.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:31.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: mon.c calling monitor election 2024-04-22T22:02:31.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:31.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:31.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:31.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:31.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T22:02:31.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: monmap e2: 2 mons at {a=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0]} 2024-04-22T22:02:31.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: fsmap 2024-04-22T22:02:31.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T22:02:31.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: mgrmap e13: y(active, since 37s) 2024-04-22T22:02:31.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: overall HEALTH_OK 2024-04-22T22:02:31.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:31.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:31.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:31.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:31.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:31.508 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-22T22:02:31.508 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"b7cbfdbc-00f3-11ef-bc93-c7b262605968","modified":"2024-04-22T22:02:26.033718Z","created":"2024-04-22T22:01:04.325314Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3301","nonce":0},{"type":"v1","addr":"172.21.15.174:6790","nonce":0}]},"addr":"172.21.15.174:6790/0","public_addr":"172.21.15.174:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T22:02:31.509 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2024-04-22T22:02:32.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:32 smithi174 ceph-mon[21825]: Deploying daemon mon.b on smithi192 2024-04-22T22:02:32.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:32 smithi174 ceph-mon[21825]: from='client.? 172.21.15.192:0/712351586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:02:32.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:32 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:32.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:32 smithi174 ceph-mon[26730]: Deploying daemon mon.b on smithi192 2024-04-22T22:02:32.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:32 smithi174 ceph-mon[26730]: from='client.? 172.21.15.192:0/712351586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:02:32.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:32 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:33.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:33 smithi174 ceph-mon[21825]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:33.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:33 smithi174 ceph-mon[26730]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:34.029 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T22:02:34.029 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:02:35.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:35 smithi174 ceph-mon[21825]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:35.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:35 smithi174 ceph-mon[26730]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:35.946 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-22T22:02:35.946 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"b7cbfdbc-00f3-11ef-bc93-c7b262605968","modified":"2024-04-22T22:02:26.033718Z","created":"2024-04-22T22:01:04.325314Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3301","nonce":0},{"type":"v1","addr":"172.21.15.174:6790","nonce":0}]},"addr":"172.21.15.174:6790/0","public_addr":"172.21.15.174:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T22:02:35.947 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2024-04-22T22:02:36.236 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:35 smithi192 podman[22779]: 2024-04-22 22:02:35.890840094 +0000 UTC m=+0.022113242 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:02:36.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 podman[22779]: 2024-04-22 22:02:36.361314242 +0000 UTC m=+0.492587371 container init 4d13e5913d298c033de53e023d3a388b37528df95532fad612644363f81b365c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-b, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, name=centos-stream, maintainer=Guillaume Abrioux , version=8, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vendor=Red Hat, Inc., ceph=True, com.redhat.component=centos-stream-container, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, RELEASE=HEAD, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T22:02:36.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 podman[22779]: 2024-04-22 22:02:36.364571447 +0000 UTC m=+0.495844577 container start 4d13e5913d298c033de53e023d3a388b37528df95532fad612644363f81b365c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-b, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, version=8, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.openshift.tags=base centos centos-stream, ceph=True, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0) 2024-04-22T22:02:36.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: set uid:gid to 167:167 (ceph:ceph) 2024-04-22T22:02:36.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-22T22:02:36.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: pidfile_write: ignore empty --pid-file 2024-04-22T22:02:36.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: load: jerasure load: lrc 2024-04-22T22:02:36.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: RocksDB version: 6.15.5 2024-04-22T22:02:36.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-22T22:02:36.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Compile date Apr 18 2022 2024-04-22T22:02:36.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: DB SUMMARY 2024-04-22T22:02:36.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: DB Session ID: A474TMTEP7H1L0H3ZDG3 2024-04-22T22:02:36.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: CURRENT file: CURRENT 2024-04-22T22:02:36.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: IDENTITY file: IDENTITY 2024-04-22T22:02:36.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-22T22:02:36.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-22T22:02:36.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-22T22:02:36.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.error_if_exists: 0 2024-04-22T22:02:36.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.create_if_missing: 0 2024-04-22T22:02:36.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.paranoid_checks: 1 2024-04-22T22:02:36.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-22T22:02:36.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.env: 0x562ad28c6860 2024-04-22T22:02:36.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.fs: Posix File System 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.info_log: 0x562ad3b8fdc0 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_file_opening_threads: 16 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.statistics: (nil) 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.use_fsync: 0 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_log_file_size: 0 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.keep_log_file_num: 1000 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.recycle_log_file_num: 0 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.allow_fallocate: 1 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.allow_mmap_reads: 0 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.allow_mmap_writes: 0 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.use_direct_reads: 0 2024-04-22T22:02:36.503 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-22T22:02:36.504 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.create_missing_column_families: 0 2024-04-22T22:02:36.504 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.db_log_dir: 2024-04-22T22:02:36.504 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-04-22T22:02:36.504 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-22T22:02:36.504 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-22T22:02:36.504 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-22T22:02:36.504 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-22T22:02:36.504 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-22T22:02:36.504 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-22T22:02:36.506 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.advise_random_on_open: 1 2024-04-22T22:02:36.506 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.db_write_buffer_size: 0 2024-04-22T22:02:36.506 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.write_buffer_manager: 0x562ad3c80240 2024-04-22T22:02:36.506 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-22T22:02:36.506 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-22T22:02:36.506 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-22T22:02:36.506 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-22T22:02:36.506 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.rate_limiter: (nil) 2024-04-22T22:02:36.506 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-22T22:02:36.506 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.wal_recovery_mode: 2 2024-04-22T22:02:36.507 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.enable_thread_tracking: 0 2024-04-22T22:02:36.507 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.enable_pipelined_write: 0 2024-04-22T22:02:36.507 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.unordered_write: 0 2024-04-22T22:02:36.507 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-22T22:02:36.507 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-22T22:02:36.507 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-22T22:02:36.507 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-22T22:02:36.507 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.row_cache: None 2024-04-22T22:02:36.507 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.wal_filter: None 2024-04-22T22:02:36.507 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-22T22:02:36.507 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.allow_ingest_behind: 0 2024-04-22T22:02:36.508 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.preserve_deletes: 0 2024-04-22T22:02:36.508 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.two_write_queues: 0 2024-04-22T22:02:36.508 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.manual_wal_flush: 0 2024-04-22T22:02:36.508 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.atomic_flush: 0 2024-04-22T22:02:36.508 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-22T22:02:36.508 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-22T22:02:36.508 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-22T22:02:36.508 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.log_readahead_size: 0 2024-04-22T22:02:36.508 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-22T22:02:36.508 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.best_efforts_recovery: 0 2024-04-22T22:02:36.509 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-22T22:02:36.509 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-22T22:02:36.509 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.allow_data_in_errors: 0 2024-04-22T22:02:36.509 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.db_host_id: __hostname__ 2024-04-22T22:02:36.509 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_background_jobs: 2 2024-04-22T22:02:36.509 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_background_compactions: -1 2024-04-22T22:02:36.509 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_subcompactions: 1 2024-04-22T22:02:36.509 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-22T22:02:36.509 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-22T22:02:36.509 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-22T22:02:36.509 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_total_wal_size: 0 2024-04-22T22:02:36.509 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-22T22:02:36.509 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-22T22:02:36.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-22T22:02:36.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-22T22:02:36.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_open_files: -1 2024-04-22T22:02:36.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.bytes_per_sync: 0 2024-04-22T22:02:36.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-22T22:02:36.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-22T22:02:36.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compaction_readahead_size: 0 2024-04-22T22:02:36.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_background_flushes: -1 2024-04-22T22:02:36.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Compression algorithms supported: 2024-04-22T22:02:36.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-22T22:02:36.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: kZSTD supported: 0 2024-04-22T22:02:36.510 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: kXpressCompression supported: 0 2024-04-22T22:02:36.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: kLZ4HCCompression supported: 1 2024-04-22T22:02:36.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: kLZ4Compression supported: 1 2024-04-22T22:02:36.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: kBZip2Compression supported: 0 2024-04-22T22:02:36.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: kZlibCompression supported: 1 2024-04-22T22:02:36.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: kSnappyCompression supported: 1 2024-04-22T22:02:36.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-22T22:02:36.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-04-22T22:02:36.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-22T22:02:36.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-22T22:02:36.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.merge_operator: 2024-04-22T22:02:36.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compaction_filter: None 2024-04-22T22:02:36.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compaction_filter_factory: None 2024-04-22T22:02:36.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.sst_partitioner_factory: None 2024-04-22T22:02:36.512 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-22T22:02:36.512 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-22T22:02:36.512 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562ad3b5bd00) 2024-04-22T22:02:36.512 INFO:journalctl@ceph.mon.b.smithi192.stdout: cache_index_and_filter_blocks: 1 2024-04-22T22:02:36.512 INFO:journalctl@ceph.mon.b.smithi192.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-22T22:02:36.512 INFO:journalctl@ceph.mon.b.smithi192.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-22T22:02:36.512 INFO:journalctl@ceph.mon.b.smithi192.stdout: pin_top_level_index_and_filter: 1 2024-04-22T22:02:36.512 INFO:journalctl@ceph.mon.b.smithi192.stdout: index_type: 0 2024-04-22T22:02:36.512 INFO:journalctl@ceph.mon.b.smithi192.stdout: data_block_index_type: 0 2024-04-22T22:02:36.512 INFO:journalctl@ceph.mon.b.smithi192.stdout: index_shortening: 1 2024-04-22T22:02:36.512 INFO:journalctl@ceph.mon.b.smithi192.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-22T22:02:36.512 INFO:journalctl@ceph.mon.b.smithi192.stdout: hash_index_allow_collision: 1 2024-04-22T22:02:36.512 INFO:journalctl@ceph.mon.b.smithi192.stdout: checksum: 1 2024-04-22T22:02:36.513 INFO:journalctl@ceph.mon.b.smithi192.stdout: no_block_cache: 0 2024-04-22T22:02:36.513 INFO:journalctl@ceph.mon.b.smithi192.stdout: block_cache: 0x562ad3bc6170 2024-04-22T22:02:36.513 INFO:journalctl@ceph.mon.b.smithi192.stdout: block_cache_name: BinnedLRUCache 2024-04-22T22:02:36.513 INFO:journalctl@ceph.mon.b.smithi192.stdout: block_cache_options: 2024-04-22T22:02:36.513 INFO:journalctl@ceph.mon.b.smithi192.stdout: capacity : 536870912 2024-04-22T22:02:36.513 INFO:journalctl@ceph.mon.b.smithi192.stdout: num_shard_bits : 4 2024-04-22T22:02:36.513 INFO:journalctl@ceph.mon.b.smithi192.stdout: strict_capacity_limit : 0 2024-04-22T22:02:36.513 INFO:journalctl@ceph.mon.b.smithi192.stdout: high_pri_pool_ratio: 0.000 2024-04-22T22:02:36.513 INFO:journalctl@ceph.mon.b.smithi192.stdout: block_cache_compressed: (nil) 2024-04-22T22:02:36.513 INFO:journalctl@ceph.mon.b.smithi192.stdout: persistent_cache: (nil) 2024-04-22T22:02:36.513 INFO:journalctl@ceph.mon.b.smithi192.stdout: block_size: 4096 2024-04-22T22:02:36.513 INFO:journalctl@ceph.mon.b.smithi192.stdout: block_size_deviation: 10 2024-04-22T22:02:36.513 INFO:journalctl@ceph.mon.b.smithi192.stdout: block_restart_interval: 16 2024-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi192.stdout: index_block_restart_interval: 1 2024-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi192.stdout: metadata_block_size: 4096 2024-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi192.stdout: partition_filters: 0 2024-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi192.stdout: use_delta_encoding: 1 2024-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi192.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi192.stdout: whole_key_filtering: 1 2024-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi192.stdout: verify_compression: 0 2024-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi192.stdout: read_amp_bytes_per_bit: 0 2024-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi192.stdout: format_version: 4 2024-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi192.stdout: enable_index_compression: 1 2024-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi192.stdout: block_align: 0 2024-04-22T22:02:36.514 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.write_buffer_size: 33554432 2024-04-22T22:02:36.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_write_buffer_number: 2 2024-04-22T22:02:36.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compression: NoCompression 2024-04-22T22:02:36.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.bottommost_compression: Disabled 2024-04-22T22:02:36.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.prefix_extractor: nullptr 2024-04-22T22:02:36.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-22T22:02:36.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.num_levels: 7 2024-04-22T22:02:36.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-22T22:02:36.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-22T22:02:36.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-22T22:02:36.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-22T22:02:36.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-22T22:02:36.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-22T22:02:36.515 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-22T22:02:36.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-22T22:02:36.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-22T22:02:36.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-22T22:02:36.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-22T22:02:36.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compression_opts.level: 32767 2024-04-22T22:02:36.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compression_opts.strategy: 0 2024-04-22T22:02:36.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-22T22:02:36.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-22T22:02:36.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-22T22:02:36.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compression_opts.enabled: false 2024-04-22T22:02:36.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-22T22:02:36.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-22T22:02:36.516 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-22T22:02:36.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.target_file_size_base: 67108864 2024-04-22T22:02:36.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-22T22:02:36.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-22T22:02:36.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-22T22:02:36.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-22T22:02:36.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-22T22:02:36.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-22T22:02:36.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-22T22:02:36.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-22T22:02:36.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-22T22:02:36.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-22T22:02:36.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-22T22:02:36.517 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-22T22:02:36.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-22T22:02:36.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.arena_block_size: 4194304 2024-04-22T22:02:36.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-22T22:02:36.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-22T22:02:36.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-22T22:02:36.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.disable_auto_compactions: 0 2024-04-22T22:02:36.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-22T22:02:36.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-22T22:02:36.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-22T22:02:36.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-22T22:02:36.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-22T22:02:36.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-22T22:02:36.518 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-22T22:02:36.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-22T22:02:36.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-22T22:02:36.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-22T22:02:36.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.table_properties_collectors: 2024-04-22T22:02:36.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.inplace_update_support: 0 2024-04-22T22:02:36.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-22T22:02:36.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-22T22:02:36.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-22T22:02:36.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-22T22:02:36.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.bloom_locality: 0 2024-04-22T22:02:36.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.max_successive_merges: 0 2024-04-22T22:02:36.519 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.paranoid_file_checks: 0 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.force_consistency_checks: 1 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.report_bg_io_stats: 0 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.ttl: 2592000 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.enable_blob_files: false 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.min_blob_size: 0 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.blob_file_size: 268435456 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-22T22:02:36.520 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-22T22:02:36.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713823356390376, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-22T22:02:36.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-22T22:02:36.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-22T22:02:36.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713823356390670, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713823356, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "894f8629-7195-49bc-bb87-5cd7e9a542c4", "db_session_id": "A474TMTEP7H1L0H3ZDG3"}} 2024-04-22T22:02:36.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-22T22:02:36.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713823356391003, "job": 1, "event": "recovery_finished"} 2024-04-22T22:02:36.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-22T22:02:36.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x562ad3baca80 2024-04-22T22:02:36.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: DB pointer 0x562ad3bbc000 2024-04-22T22:02:36.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-22T22:02:36.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-22T22:02:36.521 INFO:journalctl@ceph.mon.b.smithi192.stdout: ** DB Stats ** 2024-04-22T22:02:36.522 INFO:journalctl@ceph.mon.b.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:02:36.522 INFO:journalctl@ceph.mon.b.smithi192.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-22T22:02:36.522 INFO:journalctl@ceph.mon.b.smithi192.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-22T22:02:36.522 INFO:journalctl@ceph.mon.b.smithi192.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T22:02:36.522 INFO:journalctl@ceph.mon.b.smithi192.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-22T22:02:36.522 INFO:journalctl@ceph.mon.b.smithi192.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-22T22:02:36.522 INFO:journalctl@ceph.mon.b.smithi192.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-22T22:02:36.522 INFO:journalctl@ceph.mon.b.smithi192.stdout: 2024-04-22T22:02:36.522 INFO:journalctl@ceph.mon.b.smithi192.stdout: ** Compaction Stats [default] ** 2024-04-22T22:02:36.522 INFO:journalctl@ceph.mon.b.smithi192.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:02:36.522 INFO:journalctl@ceph.mon.b.smithi192.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:02:36.522 INFO:journalctl@ceph.mon.b.smithi192.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:36.522 INFO:journalctl@ceph.mon.b.smithi192.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:36.523 INFO:journalctl@ceph.mon.b.smithi192.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:36.523 INFO:journalctl@ceph.mon.b.smithi192.stdout: 2024-04-22T22:02:36.523 INFO:journalctl@ceph.mon.b.smithi192.stdout: ** Compaction Stats [default] ** 2024-04-22T22:02:36.523 INFO:journalctl@ceph.mon.b.smithi192.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:02:36.523 INFO:journalctl@ceph.mon.b.smithi192.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:02:36.523 INFO:journalctl@ceph.mon.b.smithi192.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:36.523 INFO:journalctl@ceph.mon.b.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:02:36.523 INFO:journalctl@ceph.mon.b.smithi192.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T22:02:36.523 INFO:journalctl@ceph.mon.b.smithi192.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T22:02:36.523 INFO:journalctl@ceph.mon.b.smithi192.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T22:02:36.523 INFO:journalctl@ceph.mon.b.smithi192.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T22:02:36.523 INFO:journalctl@ceph.mon.b.smithi192.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T22:02:36.523 INFO:journalctl@ceph.mon.b.smithi192.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:02:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:02:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T22:02:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout: 2024-04-22T22:02:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T22:02:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout: 2024-04-22T22:02:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout: ** Compaction Stats [default] ** 2024-04-22T22:02:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:02:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:02:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-22T22:02:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout: 2024-04-22T22:02:36.524 INFO:journalctl@ceph.mon.b.smithi192.stdout: ** Compaction Stats [default] ** 2024-04-22T22:02:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-22T22:02:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-22T22:02:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-22T22:02:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-22T22:02:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-22T22:02:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-22T22:02:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-22T22:02:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-22T22:02:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-22T22:02:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:02:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-22T22:02:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-22T22:02:36.525 INFO:journalctl@ceph.mon.b.smithi192.stdout: 2024-04-22T22:02:36.526 INFO:journalctl@ceph.mon.b.smithi192.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-22T22:02:36.526 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-22T22:02:36.526 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: using public_addr v2:172.21.15.192:0/0 -> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2024-04-22T22:02:36.526 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: starting mon.b rank -1 at public addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] at bind addrs [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:02:36.526 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(???) e0 preinit fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:02:36.526 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing).mds e1 new map 2024-04-22T22:02:36.526 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-22T22:02:36.526 INFO:journalctl@ceph.mon.b.smithi192.stdout: e1 2024-04-22T22:02:36.526 INFO:journalctl@ceph.mon.b.smithi192.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-22T22:02:36.526 INFO:journalctl@ceph.mon.b.smithi192.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-22T22:02:36.527 INFO:journalctl@ceph.mon.b.smithi192.stdout: legacy client fscid: -1 2024-04-22T22:02:36.527 INFO:journalctl@ceph.mon.b.smithi192.stdout: 2024-04-22T22:02:36.527 INFO:journalctl@ceph.mon.b.smithi192.stdout: No filesystems configured 2024-04-22T22:02:36.527 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-22T22:02:36.527 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-22T22:02:36.527 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-22T22:02:36.527 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-22T22:02:36.527 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-22T22:02:36.527 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-22T22:02:36.527 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-22T22:02:36.527 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:02:36.527 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:02:36.527 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-22T22:02:36.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:36.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi174:172.21.15.174=a;smithi174:[v2:172.21.15.174:3301,v1:172.21.15.174:6790]=c;smithi192:172.21.15.192=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:02:36.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: Saving service mon spec with placement smithi174:172.21.15.174=a;smithi174:[v2:172.21.15.174:3301,v1:172.21.15.174:6790]=c;smithi192:172.21.15.192=b;count:3 2024-04-22T22:02:36.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:36.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:36.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:36.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:36.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:36.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:36.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:36.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: Deploying daemon mon.c on smithi174 2024-04-22T22:02:36.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:36.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='client.? 172.21.15.192:0/1446062799' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:02:36.529 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:36.529 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:02:36.529 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.a calling monitor election 2024-04-22T22:02:36.529 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:36.529 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:36.529 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:36.529 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:36.529 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.c calling monitor election 2024-04-22T22:02:36.529 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:36.529 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:36.529 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:36.529 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:36.529 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T22:02:36.530 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: monmap e2: 2 mons at {a=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],c=[v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0]} 2024-04-22T22:02:36.530 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: fsmap 2024-04-22T22:02:36.530 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T22:02:36.530 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mgrmap e13: y(active, since 37s) 2024-04-22T22:02:36.530 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: overall HEALTH_OK 2024-04-22T22:02:36.530 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:36.530 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:36.530 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:36.530 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:36.530 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:36.530 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: Deploying daemon mon.b on smithi192 2024-04-22T22:02:36.531 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='client.? 172.21.15.192:0/712351586' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:02:36.531 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:36.531 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:36.531 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:36.531 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: from='client.? 172.21.15.192:0/3694404265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:02:36.531 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-04-22T22:02:36.531 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 ceph-mon[22835]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-22T22:02:36.531 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 bash[22779]: 4d13e5913d298c033de53e023d3a388b37528df95532fad612644363f81b365c 2024-04-22T22:02:36.531 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:36 smithi192 systemd[1]: Started Ceph mon.b for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:02:36.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:36 smithi174 ceph-mon[21825]: from='client.? 172.21.15.192:0/3694404265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:02:36.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:36 smithi174 ceph-mon[26730]: from='client.? 172.21.15.192:0/3694404265' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:02:37.931 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-22T22:02:37.931 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-22T22:02:41.670 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:02:41] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:41.670 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:02:41] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:41.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:41.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:02:41.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: mon.a calling monitor election 2024-04-22T22:02:41.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: mon.c calling monitor election 2024-04-22T22:02:41.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:41.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:41.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:41.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:41.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:41.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:41.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:41.671 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:41.672 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:41.672 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T22:02:41.672 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: monmap e3: 3 mons at {a=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0]} 2024-04-22T22:02:41.672 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: fsmap 2024-04-22T22:02:41.672 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T22:02:41.672 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: mgrmap e13: y(active, since 47s) 2024-04-22T22:02:41.672 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-22T22:02:41.672 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-22T22:02:41.673 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-22T22:02:41.673 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: mon.b (rank 2) addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] is down (out of quorum) 2024-04-22T22:02:41.673 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:41.673 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:41.673 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:41.673 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:41.673 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:41.673 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:41.673 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: mon.a calling monitor election 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: mon.c calling monitor election 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T22:02:41.674 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: monmap e3: 3 mons at {a=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0]} 2024-04-22T22:02:41.675 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: fsmap 2024-04-22T22:02:41.675 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T22:02:41.675 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: mgrmap e13: y(active, since 47s) 2024-04-22T22:02:41.675 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-22T22:02:41.675 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-22T22:02:41.675 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-22T22:02:41.675 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: mon.b (rank 2) addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] is down (out of quorum) 2024-04-22T22:02:41.675 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:41.675 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:41.675 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:41.675 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:41.675 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:41.675 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:41.814 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-04-22T22:02:41.814 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":3,"fsid":"b7cbfdbc-00f3-11ef-bc93-c7b262605968","modified":"2024-04-22T22:02:36.420515Z","created":"2024-04-22T22:01:04.325314Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3300","nonce":0},{"type":"v1","addr":"172.21.15.174:6789","nonce":0}]},"addr":"172.21.15.174:6789/0","public_addr":"172.21.15.174:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:3301","nonce":0},{"type":"v1","addr":"172.21.15.174:6790","nonce":0}]},"addr":"172.21.15.174:6790/0","public_addr":"172.21.15.174:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-22T22:02:41.816 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 3 2024-04-22T22:02:42.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[21825]: Updating smithi192:/etc/ceph/ceph.conf 2024-04-22T22:02:42.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[21825]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-22T22:02:42.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:42.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:42.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:42.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:42.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:42.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:42.567 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T22:02:42.567 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:42.567 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[21825]: from='client.? 172.21.15.192:0/1002805394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:02:42.567 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:42.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[26730]: Updating smithi192:/etc/ceph/ceph.conf 2024-04-22T22:02:42.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[26730]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-22T22:02:42.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:42.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:42.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:42.568 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:42.568 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:42.568 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:42.568 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T22:02:42.568 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:42.568 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[26730]: from='client.? 172.21.15.192:0/1002805394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:02:42.568 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:42.891 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:42 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:02:42] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:43.051 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-22T22:02:43.052 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-22T22:02:43.793 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:43.793 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:02:43.793 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: mon.a calling monitor election 2024-04-22T22:02:43.793 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: mon.c calling monitor election 2024-04-22T22:02:43.793 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:43.793 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:02:43.793 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: monmap e3: 3 mons at {a=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0]} 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: fsmap 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: mgrmap e13: y(active, since 47s) 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-22T22:02:43.794 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: mon.b (rank 2) addr [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] is down (out of quorum) 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: Updating smithi192:/etc/ceph/ceph.conf 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:43.795 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='client.? 172.21.15.192:0/1002805394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: mon.b calling monitor election 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: mon.b calling monitor election 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: mon.c calling monitor election 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: mon.a calling monitor election 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: monmap e3: 3 mons at {a=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0]} 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: fsmap 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: mgrmap e13: y(active, since 49s) 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-22T22:02:43.796 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: Cluster is now healthy 2024-04-22T22:02:43.797 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: overall HEALTH_OK 2024-04-22T22:02:43.797 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:43.797 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:43.797 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T22:02:43.797 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:43.797 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:43.797 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:44.030 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:02:43] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:44.031 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: mon.b calling monitor election 2024-04-22T22:02:44.031 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: mon.b calling monitor election 2024-04-22T22:02:44.031 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: mon.c calling monitor election 2024-04-22T22:02:44.031 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: mon.a calling monitor election 2024-04-22T22:02:44.031 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T22:02:44.031 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: monmap e3: 3 mons at {a=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0]} 2024-04-22T22:02:44.031 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: fsmap 2024-04-22T22:02:44.031 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T22:02:44.031 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: mgrmap e13: y(active, since 49s) 2024-04-22T22:02:44.031 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-22T22:02:44.031 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: Cluster is now healthy 2024-04-22T22:02:44.032 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: overall HEALTH_OK 2024-04-22T22:02:44.032 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:44.032 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:44.032 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T22:02:44.032 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:44.032 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:44.032 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:44.032 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: mon.b calling monitor election 2024-04-22T22:02:44.032 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: mon.b calling monitor election 2024-04-22T22:02:44.032 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: mon.c calling monitor election 2024-04-22T22:02:44.032 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: mon.a calling monitor election 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: monmap e3: 3 mons at {a=[v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0],b=[v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0],c=[v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0]} 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: fsmap 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: osdmap e4: 0 total, 0 up, 0 in 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: mgrmap e13: y(active, since 49s) 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: Cluster is now healthy 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: overall HEALTH_OK 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:44.033 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:43 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:44.793 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:02:44] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:45.065 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[26730]: Reconfiguring mon.c (monmap changed)... 2024-04-22T22:02:45.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[26730]: Reconfiguring daemon mon.c on smithi174 2024-04-22T22:02:45.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[26730]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:45.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:45.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:45.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:45.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:45.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[21825]: Reconfiguring mon.c (monmap changed)... 2024-04-22T22:02:45.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[21825]: Reconfiguring daemon mon.c on smithi174 2024-04-22T22:02:45.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[21825]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:45.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:45.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:45.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:45.067 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:45.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:44 smithi192 ceph-mon[22835]: Reconfiguring mon.c (monmap changed)... 2024-04-22T22:02:45.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:44 smithi192 ceph-mon[22835]: Reconfiguring daemon mon.c on smithi174 2024-04-22T22:02:45.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:44 smithi192 ceph-mon[22835]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:45.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:45.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:02:45.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:45.067 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:46.544 INFO:teuthology.orchestra.run.smithi174.stdout:# minimal ceph.conf for b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:02:46.544 INFO:teuthology.orchestra.run.smithi174.stdout:[global] 2024-04-22T22:02:46.544 INFO:teuthology.orchestra.run.smithi174.stdout: fsid = b7cbfdbc-00f3-11ef-bc93-c7b262605968 2024-04-22T22:02:46.544 INFO:teuthology.orchestra.run.smithi174.stdout: mon_host = [v2:172.21.15.174:3300/0,v1:172.21.15.174:6789/0] [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] [v2:172.21.15.174:3301/0,v1:172.21.15.174:6790/0] 2024-04-22T22:02:47.059 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:46 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:02:46] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:47.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:47 smithi192 ceph-mon[22835]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:47.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:47 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1948059733' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:47.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:47 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:47.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:47 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:47.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:47 smithi174 ceph-mon[21825]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:47.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:47 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1948059733' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:47.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:47 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:47.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:47 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:47.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:47 smithi174 ceph-mon[26730]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:47.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:47 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1948059733' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:47.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:47 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:47.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:47 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:48.002 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-22T22:02:48.003 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-22T22:02:48.003 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T22:02:48.030 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-22T22:02:48.030 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:02:48.095 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-22T22:02:48.095 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.conf 2024-04-22T22:02:48.128 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-22T22:02:48.128 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:02:48.195 INFO:tasks.cephadm:Adding mgr.y on smithi174 2024-04-22T22:02:48.195 INFO:tasks.cephadm:Adding mgr.x on smithi192 2024-04-22T22:02:48.195 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch apply mgr '2;smithi174=y;smithi192=x' 2024-04-22T22:02:49.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:49 smithi192 ceph-mon[22835]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:49.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:49 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:49.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:49 smithi192 ceph-mon[22835]: Reconfiguring mon.b (monmap changed)... 2024-04-22T22:02:49.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:49 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:49.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:49 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T22:02:49.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:49 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:49.676 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:49 smithi192 ceph-mon[22835]: Reconfiguring daemon mon.b on smithi192 2024-04-22T22:02:49.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[21825]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:49.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:49.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[21825]: Reconfiguring mon.b (monmap changed)... 2024-04-22T22:02:49.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:49.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T22:02:49.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:49.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[21825]: Reconfiguring daemon mon.b on smithi192 2024-04-22T22:02:49.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[26730]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:49.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:49.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[26730]: Reconfiguring mon.b (monmap changed)... 2024-04-22T22:02:49.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-22T22:02:49.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-22T22:02:49.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:49.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:49 smithi174 ceph-mon[26730]: Reconfiguring daemon mon.b on smithi192 2024-04-22T22:02:49.891 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mgr update... 2024-04-22T22:02:50.873 DEBUG:teuthology.orchestra.run.smithi192:mgr.x> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mgr.x.service 2024-04-22T22:02:50.875 INFO:tasks.cephadm:Deploying OSDs... 2024-04-22T22:02:50.875 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-22T22:02:50.875 DEBUG:teuthology.orchestra.run.smithi174:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T22:02:50.894 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T22:02:50.894 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_1 2024-04-22T22:02:50.952 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T22:02:50.952 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:02:50.952 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-22T22:02:50.952 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:02:50.952 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:02:50.952 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-04-22 22:02:45.322650223 +0000 2024-04-22T22:02:50.952 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-04-22 22:00:02.489386891 +0000 2024-04-22T22:02:50.952 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-04-22 22:00:02.489386891 +0000 2024-04-22T22:02:50.952 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-04-22 22:00:02.489386891 +0000 2024-04-22T22:02:50.953 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T22:02:51.017 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-04-22T22:02:51.018 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-04-22T22:02:51.018 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000270691 s, 1.9 MB/s 2024-04-22T22:02:51.019 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T22:02:51.077 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_2 2024-04-22T22:02:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T22:02:51.136 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:02:51.136 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-04-22T22:02:51.136 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:02:51.136 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:02:51.136 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-04-22 22:02:45.322650223 +0000 2024-04-22T22:02:51.137 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-04-22 22:00:02.825382425 +0000 2024-04-22T22:02:51.137 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-04-22 22:00:02.825382425 +0000 2024-04-22T22:02:51.137 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-04-22 22:00:02.825382425 +0000 2024-04-22T22:02:51.137 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T22:02:51.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:50 smithi192 ceph-mon[22835]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:51.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:50 smithi192 ceph-mon[22835]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi174=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:02:51.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:50 smithi192 ceph-mon[22835]: Saving service mgr spec with placement smithi174=y;smithi192=x;count:2 2024-04-22T22:02:51.160 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:50 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:51.198 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:50 smithi174 ceph-mon[21825]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:51.198 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:50 smithi174 ceph-mon[21825]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi174=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:02:51.198 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:50 smithi174 ceph-mon[21825]: Saving service mgr spec with placement smithi174=y;smithi192=x;count:2 2024-04-22T22:02:51.199 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:50 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:51.199 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:50 smithi174 ceph-mon[26730]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:51.199 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:50 smithi174 ceph-mon[26730]: from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi174=y;smithi192=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:02:51.199 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:50 smithi174 ceph-mon[26730]: Saving service mgr spec with placement smithi174=y;smithi192=x;count:2 2024-04-22T22:02:51.199 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:50 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:51.201 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-04-22T22:02:51.201 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-04-22T22:02:51.201 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000240618 s, 2.1 MB/s 2024-04-22T22:02:51.203 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T22:02:51.260 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_3 2024-04-22T22:02:51.316 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T22:02:51.316 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:02:51.316 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T22:02:51.316 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:02:51.316 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:02:51.316 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-04-22 22:02:45.322650223 +0000 2024-04-22T22:02:51.316 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-04-22 22:00:03.157378012 +0000 2024-04-22T22:02:51.316 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-04-22 22:00:03.157378012 +0000 2024-04-22T22:02:51.316 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-04-22 22:00:03.157378012 +0000 2024-04-22T22:02:51.317 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T22:02:51.379 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-04-22T22:02:51.379 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-04-22T22:02:51.379 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000204765 s, 2.5 MB/s 2024-04-22T22:02:51.380 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T22:02:51.436 DEBUG:teuthology.orchestra.run.smithi174:> stat /dev/vg_nvme/lv_4 2024-04-22T22:02:51.491 INFO:teuthology.orchestra.run.smithi174.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T22:02:51.491 INFO:teuthology.orchestra.run.smithi174.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:02:51.492 INFO:teuthology.orchestra.run.smithi174.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-22T22:02:51.492 INFO:teuthology.orchestra.run.smithi174.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:02:51.492 INFO:teuthology.orchestra.run.smithi174.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:02:51.492 INFO:teuthology.orchestra.run.smithi174.stdout:Access: 2024-04-22 22:02:45.322650223 +0000 2024-04-22T22:02:51.492 INFO:teuthology.orchestra.run.smithi174.stdout:Modify: 2024-04-22 22:00:03.499373466 +0000 2024-04-22T22:02:51.492 INFO:teuthology.orchestra.run.smithi174.stdout:Change: 2024-04-22 22:00:03.499373466 +0000 2024-04-22T22:02:51.492 INFO:teuthology.orchestra.run.smithi174.stdout: Birth: 2024-04-22 22:00:03.499373466 +0000 2024-04-22T22:02:51.492 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T22:02:51.553 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records in 2024-04-22T22:02:51.554 INFO:teuthology.orchestra.run.smithi174.stderr:1+0 records out 2024-04-22T22:02:51.554 INFO:teuthology.orchestra.run.smithi174.stderr:512 bytes copied, 0.000265687 s, 1.9 MB/s 2024-04-22T22:02:51.555 DEBUG:teuthology.orchestra.run.smithi174:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T22:02:51.611 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-22T22:02:51.611 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-04-22T22:02:51.626 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-22T22:02:51.626 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-04-22T22:02:51.681 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-22T22:02:51.681 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:02:51.681 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-04-22T22:02:51.681 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:02:51.681 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:02:51.681 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-22 22:02:44.002334124 +0000 2024-04-22T22:02:51.681 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-22 22:00:02.523735742 +0000 2024-04-22T22:02:51.681 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-22 22:00:02.523735742 +0000 2024-04-22T22:02:51.681 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-04-22 22:00:02.523735742 +0000 2024-04-22T22:02:51.681 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-22T22:02:51.745 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-22T22:02:51.745 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-22T22:02:51.745 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000331103 s, 1.5 MB/s 2024-04-22T22:02:51.746 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-22T22:02:51.803 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-04-22T22:02:51.858 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-22T22:02:51.859 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:02:51.859 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-22T22:02:51.859 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:02:51.859 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:02:51.859 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-22 22:02:44.002334124 +0000 2024-04-22T22:02:51.859 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-22 22:00:02.888728479 +0000 2024-04-22T22:02:51.859 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-22 22:00:02.888728479 +0000 2024-04-22T22:02:51.860 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-04-22 22:00:02.888728479 +0000 2024-04-22T22:02:51.860 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-22T22:02:51.922 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-22T22:02:51.922 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-22T22:02:51.922 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000174005 s, 2.9 MB/s 2024-04-22T22:02:51.923 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-22T22:02:51.978 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-04-22T22:02:52.032 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-22T22:02:52.032 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:02:52.033 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-22T22:02:52.033 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:02:52.033 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:02:52.033 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-22 22:02:44.002334124 +0000 2024-04-22T22:02:52.033 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-22 22:00:03.232721635 +0000 2024-04-22T22:02:52.033 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-22 22:00:03.232721635 +0000 2024-04-22T22:02:52.033 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-04-22 22:00:03.232721635 +0000 2024-04-22T22:02:52.033 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-22T22:02:52.094 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-22T22:02:52.094 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-22T22:02:52.094 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000191637 s, 2.7 MB/s 2024-04-22T22:02:52.095 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-22T22:02:52.150 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-04-22T22:02:52.205 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-22T22:02:52.205 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-22T22:02:52.205 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-04-22T22:02:52.205 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-22T22:02:52.206 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-04-22T22:02:52.206 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-04-22 22:02:44.002334124 +0000 2024-04-22T22:02:52.206 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-04-22 22:00:03.585714611 +0000 2024-04-22T22:02:52.206 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-04-22 22:00:03.585714611 +0000 2024-04-22T22:02:52.206 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-04-22 22:00:03.585714611 +0000 2024-04-22T22:02:52.206 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-22T22:02:52.267 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-04-22T22:02:52.267 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-04-22T22:02:52.267 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000187478 s, 2.7 MB/s 2024-04-22T22:02:52.268 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-22T22:02:52.323 INFO:tasks.cephadm:Deploying osd.0 on smithi174 with /dev/vg_nvme/lv_4... 2024-04-22T22:02:52.324 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T22:02:52.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:52 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:52.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:52 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:52.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:52 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:52.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:52 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:52.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:52 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:52.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:52 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T22:02:52.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:52 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T22:02:52.501 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:52 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T22:02:52.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:52 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:52.502 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:52 smithi192 ceph-mon[22835]: Deploying daemon mgr.x on smithi192 2024-04-22T22:02:52.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:52.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:52.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:52.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:52.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:52.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T22:02:52.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T22:02:52.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T22:02:52.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:52.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[21825]: Deploying daemon mgr.x on smithi192 2024-04-22T22:02:52.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:52.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:52.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:52.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:52.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:52.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T22:02:52.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-22T22:02:52.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T22:02:52.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:52.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:52 smithi174 ceph-mon[26730]: Deploying daemon mgr.x on smithi192 2024-04-22T22:02:53.059 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:52 smithi192 systemd[1]: Starting Ceph mgr.x for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:02:53.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:53 smithi192 ceph-mon[22835]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:53.316 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:53 smithi192 podman[24358]: 2024-04-22 22:02:52.985983932 +0000 UTC m=+0.018124462 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:02:53.316 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:53 smithi192 podman[24358]: 2024-04-22 22:02:53.084858799 +0000 UTC m=+0.116999336 container create bd398d520007a4d3c35cfce4ea5a36a274057d8ef0a7c5c437c407d9723a65f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, GIT_BRANCH=HEAD, vcs-type=git, architecture=x86_64, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.buildah.version=1.19.8, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-22T22:02:53.316 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:53 smithi192 podman[24358]: 2024-04-22 22:02:53.300276607 +0000 UTC m=+0.332417143 container init bd398d520007a4d3c35cfce4ea5a36a274057d8ef0a7c5c437c407d9723a65f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.tags=base centos centos-stream, release=754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, architecture=x86_64, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T22:02:53.316 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:53 smithi192 podman[24358]: 2024-04-22 22:02:53.303161332 +0000 UTC m=+0.335301870 container start bd398d520007a4d3c35cfce4ea5a36a274057d8ef0a7c5c437c407d9723a65f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, build-date=2022-05-03T08:36:31.336870, ceph=True, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., name=centos-stream, version=8, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git) 2024-04-22T22:02:53.316 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:53 smithi192 ceph-mgr[24377]: -- 172.21.15.192:0/282718582 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564cc8074340 con 0x564cc804d000 2024-04-22T22:02:53.511 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:53 smithi174 ceph-mon[21825]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:53.511 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:53 smithi174 ceph-mon[26730]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:53.655 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:53 smithi192 bash[24358]: bd398d520007a4d3c35cfce4ea5a36a274057d8ef0a7c5c437c407d9723a65f1 2024-04-22T22:02:53.655 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:53 smithi192 systemd[1]: Started Ceph mgr.x for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:02:53.655 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:53 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:53.448+0000 7f27b72c9000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T22:02:53.655 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:53 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:53.582+0000 7f27b72c9000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T22:02:53.655 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:53 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:53.678+0000 7f27b72c9000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T22:02:54.200 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:53.999+0000 7f27b72c9000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T22:02:54.200 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:54.083+0000 7f27b72c9000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T22:02:54.355 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T22:02:54.355 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T22:02:54.355 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-04-22T22:02:54.356 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-04-22T22:02:54.356 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112704 s, 930 MB/s 2024-04-22T22:02:54.356 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-04-22T22:02:54.762 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:54.762 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:54.762 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:54.762 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:54.762 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:54.762 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:02:54.762 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: Reconfiguring mgr.y (unknown last config time)... 2024-04-22T22:02:54.762 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T22:02:54.762 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T22:02:54.763 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:54.763 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: Reconfiguring daemon mgr.y on smithi174 2024-04-22T22:02:54.763 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:02:54.763 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:02:54.763 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:54.763 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:54.763 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:54.763 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:54.763 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:54.763 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:54.763 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:02:54.763 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: Reconfiguring mgr.y (unknown last config time)... 2024-04-22T22:02:54.763 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T22:02:54.764 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T22:02:54.764 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:54.764 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: Reconfiguring daemon mgr.y on smithi174 2024-04-22T22:02:54.764 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:02:54.764 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:02:54.764 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:54.799 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:54.799 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:54.799 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:54.799 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:54.799 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:54.800 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:02:54.800 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: Reconfiguring mgr.y (unknown last config time)... 2024-04-22T22:02:54.800 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-22T22:02:54.800 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-22T22:02:54.800 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:54.800 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: Reconfiguring daemon mgr.y on smithi174 2024-04-22T22:02:54.800 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:02:54.800 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:02:54.800 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:55.315 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:55.285+0000 7f27b72c9000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T22:02:55.683 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:55 smithi174 ceph-mon[21825]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:55.683 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:55.683 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:55.683 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:55.683 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:55.683 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:55.683 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:55 smithi174 ceph-mon[26730]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:55.683 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:55.683 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:55.683 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:55.684 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:55.684 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:55.709 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_4 2024-04-22T22:02:55.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:55 smithi192 ceph-mon[22835]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:55.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:55.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:55.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:55.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:55.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:55.816 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:55.466+0000 7f27b72c9000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T22:02:55.816 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:55.551+0000 7f27b72c9000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T22:02:56.205 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:55.886+0000 7f27b72c9000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T22:02:56.206 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:55.976+0000 7f27b72c9000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T22:02:56.206 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:56 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:56.069+0000 7f27b72c9000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T22:02:56.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:02:56] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:56.472 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:56 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:56.324+0000 7f27b72c9000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T22:02:56.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:56 smithi192 ceph-mon[22835]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:02:56.742 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:56 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:56.743 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:56 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:56.743 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:56 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:56.743 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:56 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:56.743 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:56 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:56.743 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:56 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:56.815 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:02:56] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:02:56.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[21825]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:02:56.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:56.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:56.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:56.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:56.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:56.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:56.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[26730]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:02:56.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:56.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:56.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:02:56.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:56.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:02:56.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:56 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:57.065 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:56 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:56.765+0000 7f27b72c9000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T22:02:57.489 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:57 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:57.247+0000 7f27b72c9000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T22:02:57.489 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:57 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:57.338+0000 7f27b72c9000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T22:02:57.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:57 smithi192 ceph-mon[22835]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:57.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:57 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:57.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:57 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:57.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:57 smithi174 ceph-mon[21825]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:57.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:57 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:57.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:57 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:57.817 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:57 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:57.625+0000 7f27b72c9000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T22:02:57.817 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:57 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:57.713+0000 7f27b72c9000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T22:02:57.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:57 smithi174 ceph-mon[26730]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:57.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:57 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:57.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:57 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:02:58.424 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:58 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:58.116+0000 7f27b72c9000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T22:02:58.424 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:58 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:58.246+0000 7f27b72c9000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T22:02:58.424 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:58 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:58.336+0000 7f27b72c9000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T22:02:58.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:02:58.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:02:58.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:58.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:02:58.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:02:58.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:58.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:02:58.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:02:58.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:02:58.816 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:58 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:58.447+0000 7f27b72c9000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T22:02:58.816 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:58 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:58.535+0000 7f27b72c9000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T22:02:59.315 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:59 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:59.070+0000 7f27b72c9000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T22:02:59.815 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:02:59 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:02:59.430+0000 7f27b72c9000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T22:02:59.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:59 smithi192 ceph-mon[22835]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:02:59.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:59 smithi192 ceph-mon[22835]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:59.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:59 smithi192 ceph-mon[22835]: Standby manager daemon x started 2024-04-22T22:02:59.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:59 smithi192 ceph-mon[22835]: from='mgr.? 172.21.15.192:0/2085747254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-22T22:02:59.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:59 smithi192 ceph-mon[22835]: from='mgr.? 172.21.15.192:0/2085747254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T22:02:59.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:59 smithi192 ceph-mon[22835]: from='mgr.? 172.21.15.192:0/2085747254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-22T22:02:59.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:02:59 smithi192 ceph-mon[22835]: from='mgr.? 172.21.15.192:0/2085747254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T22:02:59.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[21825]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:02:59.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[21825]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:59.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[21825]: Standby manager daemon x started 2024-04-22T22:02:59.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[21825]: from='mgr.? 172.21.15.192:0/2085747254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-22T22:02:59.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[21825]: from='mgr.? 172.21.15.192:0/2085747254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T22:02:59.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[21825]: from='mgr.? 172.21.15.192:0/2085747254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-22T22:02:59.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[21825]: from='mgr.? 172.21.15.192:0/2085747254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T22:02:59.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[26730]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:02:59.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[26730]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:02:59.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[26730]: Standby manager daemon x started 2024-04-22T22:02:59.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[26730]: from='mgr.? 172.21.15.192:0/2085747254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-22T22:02:59.818 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[26730]: from='mgr.? 172.21.15.192:0/2085747254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T22:02:59.818 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[26730]: from='mgr.? 172.21.15.192:0/2085747254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-22T22:02:59.818 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:02:59 smithi174 ceph-mon[26730]: from='mgr.? 172.21.15.192:0/2085747254' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T22:03:00.788 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:00 smithi174 ceph-mon[21825]: mgrmap e14: y(active, since 65s), standbys: x 2024-04-22T22:03:00.789 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:00 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T22:03:00.789 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:00 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3476316209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10b65a80-27ba-4f98-b4a5-46610cbcee53"}]: dispatch 2024-04-22T22:03:00.789 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:00 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3476316209' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "10b65a80-27ba-4f98-b4a5-46610cbcee53"}]': finished 2024-04-22T22:03:00.789 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:00 smithi174 ceph-mon[21825]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T22:03:00.789 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:00 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:00.789 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:00 smithi174 ceph-mon[26730]: mgrmap e14: y(active, since 65s), standbys: x 2024-04-22T22:03:00.789 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:00 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T22:03:00.789 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:00 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3476316209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10b65a80-27ba-4f98-b4a5-46610cbcee53"}]: dispatch 2024-04-22T22:03:00.789 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:00 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3476316209' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "10b65a80-27ba-4f98-b4a5-46610cbcee53"}]': finished 2024-04-22T22:03:00.790 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:00 smithi174 ceph-mon[26730]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T22:03:00.790 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:00 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:00.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:00 smithi192 ceph-mon[22835]: mgrmap e14: y(active, since 65s), standbys: x 2024-04-22T22:03:00.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:00 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T22:03:00.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:00 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3476316209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "10b65a80-27ba-4f98-b4a5-46610cbcee53"}]: dispatch 2024-04-22T22:03:00.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:00 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3476316209' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "10b65a80-27ba-4f98-b4a5-46610cbcee53"}]': finished 2024-04-22T22:03:00.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:00 smithi192 ceph-mon[22835]: osdmap e5: 1 total, 0 up, 1 in 2024-04-22T22:03:00.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:00 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:01.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:01 smithi174 ceph-mon[26730]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:01.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:01 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/4272202905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:03:01.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:01 smithi174 ceph-mon[21825]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:01.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:01 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/4272202905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:03:01.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:01 smithi192 ceph-mon[22835]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:01.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:01 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/4272202905' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:03:03.009 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:03 smithi174 ceph-mon[21825]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:03.009 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:03 smithi174 ceph-mon[26730]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:03.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:03 smithi192 ceph-mon[22835]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:05.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:05 smithi192 ceph-mon[22835]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:05.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:05 smithi174 ceph-mon[26730]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:05.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:05 smithi174 ceph-mon[21825]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:06.178 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:03:05 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:03:05] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:03:07.269 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:06 smithi174 ceph-mon[26730]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:07.269 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:06 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:07.269 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:06 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:07.269 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:06 smithi174 ceph-mon[21825]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:07.270 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:06 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:07.270 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:06 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:07.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:06 smithi192 ceph-mon[22835]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:07.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:06 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:07.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:06 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:08.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:08 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T22:03:08.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:08 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:08.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:08 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T22:03:08.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:08 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:08.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:08 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-22T22:03:08.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:08 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:09.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:09 smithi174 ceph-mon[21825]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:09.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:09 smithi174 ceph-mon[21825]: Deploying daemon osd.0 on smithi174 2024-04-22T22:03:09.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:09 smithi192 ceph-mon[22835]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:09.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:09 smithi192 ceph-mon[22835]: Deploying daemon osd.0 on smithi174 2024-04-22T22:03:09.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:09 smithi174 ceph-mon[26730]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:09.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:09 smithi174 ceph-mon[26730]: Deploying daemon osd.0 on smithi174 2024-04-22T22:03:11.227 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:11 smithi174 ceph-mon[21825]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:11.227 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:11 smithi174 ceph-mon[26730]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:11.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:11 smithi192 ceph-mon[22835]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:13.241 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:13 smithi174 ceph-mon[21825]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:13.241 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:13 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:13.241 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:13 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:13.241 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:13 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:13.241 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:13 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:13.241 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:13 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:13.242 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:13 smithi174 ceph-mon[26730]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:13.242 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:13 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:13.242 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:13 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:13.242 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:13 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:13.242 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:13 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:13.242 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:13 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:13.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:13 smithi192 ceph-mon[22835]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:13.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:13 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:13.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:13 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:13.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:13 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:13.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:13 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:13.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:13 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:14.566 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:03:14 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:03:14] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:03:15.462 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:15 smithi174 ceph-mon[21825]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:15.462 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:15 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:15.462 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:15 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:15.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:15 smithi174 ceph-mon[26730]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:15.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:15 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:15.463 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:15 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:15.497 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 0 on host 'smithi174' 2024-04-22T22:03:15.808 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:15 smithi192 ceph-mon[22835]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:15.809 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:15 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:15.809 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:15 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:16.156 DEBUG:teuthology.orchestra.run.smithi174:osd.0> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.0.service 2024-04-22T22:03:16.159 INFO:tasks.cephadm:Deploying osd.1 on smithi174 with /dev/vg_nvme/lv_3... 2024-04-22T22:03:16.159 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T22:03:16.761 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:16.761 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:16.761 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:16.761 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:16.761 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:03:16.761 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:16.761 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:16.761 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[21825]: from='osd.0 [v2:172.21.15.174:6802/2571929691,v1:172.21.15.174:6803/2571929691]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T22:03:16.761 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[21825]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T22:03:16.762 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:16.762 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:16.762 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:16.762 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:16.762 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:03:16.762 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:16.762 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:16.762 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[26730]: from='osd.0 [v2:172.21.15.174:6802/2571929691,v1:172.21.15.174:6803/2571929691]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T22:03:16.762 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:16 smithi174 ceph-mon[26730]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T22:03:16.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:16.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:16.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:16.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:16.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:03:16.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:16.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:16.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:16 smithi192 ceph-mon[22835]: from='osd.0 [v2:172.21.15.174:6802/2571929691,v1:172.21.15.174:6803/2571929691]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T22:03:16.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:16 smithi192 ceph-mon[22835]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-22T22:03:17.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:17 smithi192 ceph-mon[22835]: Adjusting osd_memory_target on smithi174 to 15040M 2024-04-22T22:03:17.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:17 smithi192 ceph-mon[22835]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:17.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:17 smithi192 ceph-mon[22835]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T22:03:17.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:17 smithi192 ceph-mon[22835]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T22:03:17.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:17 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:17.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:17 smithi192 ceph-mon[22835]: from='osd.0 [v2:172.21.15.174:6802/2571929691,v1:172.21.15.174:6803/2571929691]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:03:17.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:17 smithi192 ceph-mon[22835]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:03:17.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[21825]: Adjusting osd_memory_target on smithi174 to 15040M 2024-04-22T22:03:17.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[21825]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:17.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[21825]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T22:03:17.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[21825]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T22:03:17.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:17.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[21825]: from='osd.0 [v2:172.21.15.174:6802/2571929691,v1:172.21.15.174:6803/2571929691]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:03:17.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[21825]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:03:17.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[26730]: Adjusting osd_memory_target on smithi174 to 15040M 2024-04-22T22:03:17.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[26730]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:17.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[26730]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-22T22:03:17.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[26730]: osdmap e6: 1 total, 0 up, 1 in 2024-04-22T22:03:17.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:17.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[26730]: from='osd.0 [v2:172.21.15.174:6802/2571929691,v1:172.21.15.174:6803/2571929691]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:03:17.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-mon[26730]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:03:17.818 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0[29984]: 2024-04-22T22:03:17.558+0000 7f3b0fb35700 -1 osd.0 0 waiting for initial osdmap 2024-04-22T22:03:17.818 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:03:17 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0[29984]: 2024-04-22T22:03:17.560+0000 7f3b0accd700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T22:03:18.093 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T22:03:18.093 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T22:03:18.094 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-04-22T22:03:18.094 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-04-22T22:03:18.094 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127119 s, 825 MB/s 2024-04-22T22:03:18.094 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-04-22T22:03:18.688 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:18 smithi174 ceph-mon[21825]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-22T22:03:18.688 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:18 smithi174 ceph-mon[21825]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T22:03:18.688 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:18 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:18.688 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:18 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:18.688 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:18 smithi174 ceph-mon[21825]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:18.688 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:18 smithi174 ceph-mon[26730]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-22T22:03:18.688 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:18 smithi174 ceph-mon[26730]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T22:03:18.688 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:18 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:18.688 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:18 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:18.689 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:18 smithi174 ceph-mon[26730]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:18.712 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_3 2024-04-22T22:03:18.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:18 smithi192 ceph-mon[22835]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-22T22:03:18.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:18 smithi192 ceph-mon[22835]: osdmap e7: 1 total, 0 up, 1 in 2024-04-22T22:03:18.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:18 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:18.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:18 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:18.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:18 smithi192 ceph-mon[22835]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-22T22:03:19.819 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:19 smithi192 ceph-mon[22835]: purged_snaps scrub starts 2024-04-22T22:03:19.819 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:19 smithi192 ceph-mon[22835]: purged_snaps scrub ok 2024-04-22T22:03:19.820 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:19 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:19.820 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:19 smithi192 ceph-mon[22835]: osd.0 [v2:172.21.15.174:6802/2571929691,v1:172.21.15.174:6803/2571929691] boot 2024-04-22T22:03:19.820 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:19 smithi192 ceph-mon[22835]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T22:03:19.820 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:19 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:19.820 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:19 smithi174 ceph-mon[21825]: purged_snaps scrub starts 2024-04-22T22:03:19.821 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:19 smithi174 ceph-mon[21825]: purged_snaps scrub ok 2024-04-22T22:03:19.821 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:19 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:19.821 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:19 smithi174 ceph-mon[21825]: osd.0 [v2:172.21.15.174:6802/2571929691,v1:172.21.15.174:6803/2571929691] boot 2024-04-22T22:03:19.821 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:19 smithi174 ceph-mon[21825]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T22:03:19.821 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:19 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:19.822 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:19 smithi174 ceph-mon[26730]: purged_snaps scrub starts 2024-04-22T22:03:19.822 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:19 smithi174 ceph-mon[26730]: purged_snaps scrub ok 2024-04-22T22:03:19.822 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:19 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:19.822 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:19 smithi174 ceph-mon[26730]: osd.0 [v2:172.21.15.174:6802/2571929691,v1:172.21.15.174:6803/2571929691] boot 2024-04-22T22:03:19.822 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:19 smithi174 ceph-mon[26730]: osdmap e8: 1 total, 1 up, 1 in 2024-04-22T22:03:19.823 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:19 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:03:20.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:20 smithi174 ceph-mon[21825]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T22:03:20.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:20 smithi174 ceph-mon[21825]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:20.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:20 smithi174 ceph-mon[26730]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T22:03:20.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:20 smithi174 ceph-mon[26730]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:20.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:20 smithi192 ceph-mon[22835]: osdmap e9: 1 total, 1 up, 1 in 2024-04-22T22:03:20.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:20 smithi192 ceph-mon[22835]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:21.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:21 smithi174 ceph-mon[21825]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:03:21.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:21 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:03:21.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:21 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:03:21.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:21 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:21.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:21 smithi174 ceph-mon[26730]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:03:21.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:21 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:03:21.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:21 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:03:21.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:21 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:21.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:21 smithi192 ceph-mon[22835]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:03:21.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:21 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:03:21.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:21 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:03:21.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:21 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:23.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:23 smithi174 ceph-mon[21825]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:23.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:23 smithi174 ceph-mon[26730]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:23.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:23 smithi192 ceph-mon[22835]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:23.816 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:03:23 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:03:23] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:03:24.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:24 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:24.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:24 smithi192 ceph-mon[22835]: Detected new or changed devices on smithi174 2024-04-22T22:03:24.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:24 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:24.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:24 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/559726076' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4d7e7fc-0a42-4a61-ba4f-a1693dc2a970"}]: dispatch 2024-04-22T22:03:24.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:24 smithi192 ceph-mon[22835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4d7e7fc-0a42-4a61-ba4f-a1693dc2a970"}]: dispatch 2024-04-22T22:03:24.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:24 smithi192 ceph-mon[22835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4d7e7fc-0a42-4a61-ba4f-a1693dc2a970"}]': finished 2024-04-22T22:03:24.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:24 smithi192 ceph-mon[22835]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T22:03:24.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:24 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:24.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:24.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[21825]: Detected new or changed devices on smithi174 2024-04-22T22:03:24.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:24.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/559726076' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4d7e7fc-0a42-4a61-ba4f-a1693dc2a970"}]: dispatch 2024-04-22T22:03:24.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[21825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4d7e7fc-0a42-4a61-ba4f-a1693dc2a970"}]: dispatch 2024-04-22T22:03:24.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[21825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4d7e7fc-0a42-4a61-ba4f-a1693dc2a970"}]': finished 2024-04-22T22:03:24.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[21825]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T22:03:24.817 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:24.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:24.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[26730]: Detected new or changed devices on smithi174 2024-04-22T22:03:24.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:24.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/559726076' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4d7e7fc-0a42-4a61-ba4f-a1693dc2a970"}]: dispatch 2024-04-22T22:03:24.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[26730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4d7e7fc-0a42-4a61-ba4f-a1693dc2a970"}]: dispatch 2024-04-22T22:03:24.818 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[26730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4d7e7fc-0a42-4a61-ba4f-a1693dc2a970"}]': finished 2024-04-22T22:03:24.818 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[26730]: osdmap e10: 2 total, 1 up, 2 in 2024-04-22T22:03:24.818 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:24 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:25.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:25 smithi192 ceph-mon[22835]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:25.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:25 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2237894960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:03:25.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:25 smithi174 ceph-mon[21825]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:25.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:25 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2237894960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:03:25.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:25 smithi174 ceph-mon[26730]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:25.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:25 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2237894960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:03:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:27 smithi174 ceph-mon[21825]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:27.796 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:27 smithi174 ceph-mon[26730]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:27.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:27 smithi192 ceph-mon[22835]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:29.779 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:29 smithi174 ceph-mon[21825]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:29.780 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:29 smithi174 ceph-mon[26730]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:29.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:29 smithi192 ceph-mon[22835]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:30.654 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:30 smithi174 ceph-mon[21825]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:30.654 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:30 smithi174 ceph-mon[26730]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:30.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:30 smithi192 ceph-mon[22835]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:31.592 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T22:03:31.592 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:31.592 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:31 smithi174 ceph-mon[21825]: Deploying daemon osd.1 on smithi174 2024-04-22T22:03:31.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T22:03:31.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:31.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:31 smithi174 ceph-mon[26730]: Deploying daemon osd.1 on smithi174 2024-04-22T22:03:31.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:31 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-22T22:03:31.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:31 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:31.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:31 smithi192 ceph-mon[22835]: Deploying daemon osd.1 on smithi174 2024-04-22T22:03:33.313 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:33 smithi192 ceph-mon[22835]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:33.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:33 smithi174 ceph-mon[21825]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:33.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:33 smithi174 ceph-mon[26730]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:34.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:03:34 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:03:34] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:03:35.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:35 smithi192 ceph-mon[22835]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:35.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:35 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:35.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:35 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:35.752 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:35 smithi174 ceph-mon[21825]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:35.752 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:35 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:35.752 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:35 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:35.753 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:35 smithi174 ceph-mon[26730]: pgmap v50: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:35.753 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:35 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:35.753 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:35 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:37.258 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:37 smithi174 ceph-mon[26730]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:37.258 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:37 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:37.258 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:37 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:37.258 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:37 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:37.259 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:37 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:37.259 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:37 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:37.259 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:37 smithi174 ceph-mon[21825]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:37.259 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:37 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:37.259 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:37 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:37.259 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:37 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:37.259 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:37 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:37.259 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:37 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:37.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:37 smithi192 ceph-mon[22835]: pgmap v51: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:37.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:37 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:37.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:37 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:37.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:37 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:37.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:37 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:37.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:37 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:38.646 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 1 on host 'smithi174' 2024-04-22T22:03:39.267 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[21825]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:39.268 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:39.268 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:39.268 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:39.268 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:39.268 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:03:39.268 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:03:39.268 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:39.268 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:39.268 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[26730]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:39.269 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:39.269 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:39.269 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:39.269 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:39.269 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:03:39.269 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:03:39.269 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:39.269 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:39 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:39.289 DEBUG:teuthology.orchestra.run.smithi174:osd.1> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.1.service 2024-04-22T22:03:39.292 INFO:tasks.cephadm:Deploying osd.2 on smithi174 with /dev/vg_nvme/lv_2... 2024-04-22T22:03:39.292 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T22:03:39.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:39 smithi192 ceph-mon[22835]: pgmap v52: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:39.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:39 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:39.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:39 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:39.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:39 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:39.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:39 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:39.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:39 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:03:39.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:39 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:03:39.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:39 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:39.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:39 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:40.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:40 smithi174 ceph-mon[21825]: Adjusting osd_memory_target on smithi174 to 7520M 2024-04-22T22:03:40.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:40 smithi174 ceph-mon[21825]: from='osd.1 [v2:172.21.15.174:6810/222655237,v1:172.21.15.174:6811/222655237]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T22:03:40.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:40 smithi174 ceph-mon[21825]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T22:03:40.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:40 smithi174 ceph-mon[26730]: Adjusting osd_memory_target on smithi174 to 7520M 2024-04-22T22:03:40.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:40 smithi174 ceph-mon[26730]: from='osd.1 [v2:172.21.15.174:6810/222655237,v1:172.21.15.174:6811/222655237]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T22:03:40.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:40 smithi174 ceph-mon[26730]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T22:03:40.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:40 smithi192 ceph-mon[22835]: Adjusting osd_memory_target on smithi174 to 7520M 2024-04-22T22:03:40.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:40 smithi192 ceph-mon[22835]: from='osd.1 [v2:172.21.15.174:6810/222655237,v1:172.21.15.174:6811/222655237]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T22:03:40.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:40 smithi192 ceph-mon[22835]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-22T22:03:41.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:41 smithi192 ceph-mon[22835]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:41.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:41 smithi192 ceph-mon[22835]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T22:03:41.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:41 smithi192 ceph-mon[22835]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T22:03:41.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:41 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:41.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:41 smithi192 ceph-mon[22835]: from='osd.1 [v2:172.21.15.174:6810/222655237,v1:172.21.15.174:6811/222655237]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:03:41.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:41 smithi192 ceph-mon[22835]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:03:41.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-mon[21825]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:41.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-mon[21825]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T22:03:41.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-mon[21825]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T22:03:41.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:41.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-mon[21825]: from='osd.1 [v2:172.21.15.174:6810/222655237,v1:172.21.15.174:6811/222655237]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:03:41.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-mon[21825]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:03:41.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-mon[26730]: pgmap v53: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:41.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-mon[26730]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-22T22:03:41.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-mon[26730]: osdmap e11: 2 total, 1 up, 2 in 2024-04-22T22:03:41.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:41.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-mon[26730]: from='osd.1 [v2:172.21.15.174:6810/222655237,v1:172.21.15.174:6811/222655237]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:03:41.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-mon[26730]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:03:41.318 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1[32874]: 2024-04-22T22:03:41.029+0000 7fc8c539e700 -1 osd.1 0 waiting for initial osdmap 2024-04-22T22:03:41.318 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:03:41 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1[32874]: 2024-04-22T22:03:41.031+0000 7fc8c1d39700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T22:03:41.362 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T22:03:41.363 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T22:03:41.363 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-04-22T22:03:41.363 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-04-22T22:03:41.363 INFO:teuthology.orchestra.run.smithi174.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140166 s, 748 MB/s 2024-04-22T22:03:41.363 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-04-22T22:03:41.991 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_2 2024-04-22T22:03:42.297 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:42 smithi174 ceph-mon[26730]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-22T22:03:42.297 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:42 smithi174 ceph-mon[26730]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T22:03:42.297 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:42 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:42.297 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:42 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:42.297 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:42 smithi174 ceph-mon[21825]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-22T22:03:42.297 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:42 smithi174 ceph-mon[21825]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T22:03:42.298 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:42 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:42.298 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:42 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:42.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:42 smithi192 ceph-mon[22835]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-22T22:03:42.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:42 smithi192 ceph-mon[22835]: osdmap e12: 2 total, 1 up, 2 in 2024-04-22T22:03:42.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:42 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:42.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:42 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:43.300 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[26730]: purged_snaps scrub starts 2024-04-22T22:03:43.301 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[26730]: purged_snaps scrub ok 2024-04-22T22:03:43.301 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[26730]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:43.301 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:43.301 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[26730]: osd.1 [v2:172.21.15.174:6810/222655237,v1:172.21.15.174:6811/222655237] boot 2024-04-22T22:03:43.301 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[26730]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T22:03:43.301 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:43.301 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[21825]: purged_snaps scrub starts 2024-04-22T22:03:43.301 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[21825]: purged_snaps scrub ok 2024-04-22T22:03:43.301 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[21825]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:43.301 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:43.301 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[21825]: osd.1 [v2:172.21.15.174:6810/222655237,v1:172.21.15.174:6811/222655237] boot 2024-04-22T22:03:43.302 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[21825]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T22:03:43.302 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:43.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:43 smithi192 ceph-mon[22835]: purged_snaps scrub starts 2024-04-22T22:03:43.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:43 smithi192 ceph-mon[22835]: purged_snaps scrub ok 2024-04-22T22:03:43.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:43 smithi192 ceph-mon[22835]: pgmap v56: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-22T22:03:43.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:43.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:43 smithi192 ceph-mon[22835]: osd.1 [v2:172.21.15.174:6810/222655237,v1:172.21.15.174:6811/222655237] boot 2024-04-22T22:03:43.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:43 smithi192 ceph-mon[22835]: osdmap e13: 2 total, 2 up, 2 in 2024-04-22T22:03:43.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:43 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:03:43.846 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:03:43 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:03:43] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:03:44.162 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[21825]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T22:03:44.162 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:44.162 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:44.162 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:44.162 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:44.162 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:44.162 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:44.162 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:03:44.163 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:03:44.163 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:44.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[26730]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T22:03:44.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:44.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:44.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:44.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:44.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:44.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:44.163 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:03:44.164 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:03:44.164 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:44.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:44 smithi192 ceph-mon[22835]: osdmap e14: 2 total, 2 up, 2 in 2024-04-22T22:03:44.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:44.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:44.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:44.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:44.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:44.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:44.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:03:44.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:03:44.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:45.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:45 smithi192 ceph-mon[22835]: Detected new or changed devices on smithi174 2024-04-22T22:03:45.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:45 smithi192 ceph-mon[22835]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:45.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:45 smithi192 ceph-mon[22835]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:03:45.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:45 smithi192 ceph-mon[22835]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-22T22:03:45.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:45 smithi174 ceph-mon[21825]: Detected new or changed devices on smithi174 2024-04-22T22:03:45.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:45 smithi174 ceph-mon[21825]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:45.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:45 smithi174 ceph-mon[21825]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:03:45.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:45 smithi174 ceph-mon[21825]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-22T22:03:45.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:45 smithi174 ceph-mon[26730]: Detected new or changed devices on smithi174 2024-04-22T22:03:45.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:45 smithi174 ceph-mon[26730]: pgmap v59: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:45.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:45 smithi174 ceph-mon[26730]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:03:45.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:45 smithi174 ceph-mon[26730]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-22T22:03:47.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:47 smithi174 ceph-mon[21825]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:47.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:47 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2453126815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "439d2d85-e8b4-4a98-bc6d-11fcab19c75d"}]: dispatch 2024-04-22T22:03:47.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:47 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2453126815' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "439d2d85-e8b4-4a98-bc6d-11fcab19c75d"}]': finished 2024-04-22T22:03:47.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:47 smithi174 ceph-mon[21825]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T22:03:47.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:47 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:03:47.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:47 smithi174 ceph-mon[26730]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:47.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:47 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2453126815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "439d2d85-e8b4-4a98-bc6d-11fcab19c75d"}]: dispatch 2024-04-22T22:03:47.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:47 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2453126815' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "439d2d85-e8b4-4a98-bc6d-11fcab19c75d"}]': finished 2024-04-22T22:03:47.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:47 smithi174 ceph-mon[26730]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T22:03:47.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:47 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:03:47.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:47 smithi192 ceph-mon[22835]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:47.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:47 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2453126815' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "439d2d85-e8b4-4a98-bc6d-11fcab19c75d"}]: dispatch 2024-04-22T22:03:47.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:47 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2453126815' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "439d2d85-e8b4-4a98-bc6d-11fcab19c75d"}]': finished 2024-04-22T22:03:47.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:47 smithi192 ceph-mon[22835]: osdmap e15: 3 total, 2 up, 3 in 2024-04-22T22:03:47.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:47 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:03:47.816 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:03:47 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:03:47] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:03:48.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:48 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/4053932842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:03:48.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:48 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:48.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:48 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:48.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:48 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:48.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:48 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:48.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:48 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:48.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:48 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:48.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/4053932842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:03:48.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:48.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:48.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:48.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:48.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:48.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:48.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/4053932842' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:03:48.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:48.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:48.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:03:48.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:48.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:03:48.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:48 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:49.319 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:49 smithi192 ceph-mon[22835]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:49.319 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:49 smithi192 ceph-mon[22835]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-22T22:03:49.319 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:49 smithi192 ceph-mon[22835]: Cluster is now healthy 2024-04-22T22:03:49.319 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:49 smithi174 ceph-mon[21825]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:49.319 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:49 smithi174 ceph-mon[21825]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-22T22:03:49.319 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:49 smithi174 ceph-mon[21825]: Cluster is now healthy 2024-04-22T22:03:49.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:49 smithi174 ceph-mon[26730]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:49.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:49 smithi174 ceph-mon[26730]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-22T22:03:49.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:49 smithi174 ceph-mon[26730]: Cluster is now healthy 2024-04-22T22:03:51.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:51 smithi192 ceph-mon[22835]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:51.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:51 smithi174 ceph-mon[21825]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:51.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:51 smithi174 ceph-mon[26730]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:53.227 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:53 smithi174 ceph-mon[21825]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:53.227 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:53 smithi174 ceph-mon[26730]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:53.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:53 smithi192 ceph-mon[22835]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:54.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:03:54.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:03:54.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T22:03:54.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:54.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:03:54.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:03:54.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T22:03:54.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:54.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:03:54.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:03:54.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-22T22:03:54.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:03:54.657 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:03:54 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:03:54] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:03:55.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:55 smithi174 ceph-mon[26730]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:55.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:55 smithi174 ceph-mon[26730]: Deploying daemon osd.2 on smithi174 2024-04-22T22:03:55.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:55.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:55.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:55 smithi174 ceph-mon[21825]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:55.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:55 smithi174 ceph-mon[21825]: Deploying daemon osd.2 on smithi174 2024-04-22T22:03:55.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:55.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:55.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:55 smithi192 ceph-mon[22835]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:55.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:55 smithi192 ceph-mon[22835]: Deploying daemon osd.2 on smithi174 2024-04-22T22:03:55.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:55.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:03:57.291 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:57 smithi174 ceph-mon[21825]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:57.291 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:57 smithi174 ceph-mon[26730]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:57.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:57 smithi192 ceph-mon[22835]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:59.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:03:59 smithi174 ceph-mon[21825]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:59.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:03:59 smithi174 ceph-mon[26730]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:03:59.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:03:59 smithi192 ceph-mon[22835]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:04:00.441 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:00 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:00.442 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:00 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:00.442 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:00 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:00.442 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:00 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:00.442 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:00 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:00.442 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:00 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:00.442 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:00 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:00.442 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:00 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:00.442 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:00 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:00.442 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:00 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:00.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:00 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:00.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:00 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:00.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:00 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:00.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:00 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:00.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:00 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:01.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:01 smithi174 ceph-mon[21825]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:04:01.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:01 smithi174 ceph-mon[26730]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:04:01.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:01 smithi192 ceph-mon[22835]: pgmap v68: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:04:01.986 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 2 on host 'smithi174' 2024-04-22T22:04:02.724 DEBUG:teuthology.orchestra.run.smithi174:osd.2> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.2.service 2024-04-22T22:04:02.727 INFO:tasks.cephadm:Deploying osd.3 on smithi174 with /dev/vg_nvme/lv_1... 2024-04-22T22:04:02.727 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T22:04:03.297 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[21825]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:04:03.297 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:03.297 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:03.297 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:03.297 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:03.297 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:03.297 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:03.297 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:03.297 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[21825]: Adjusting osd_memory_target on smithi174 to 5013M 2024-04-22T22:04:03.297 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:03.298 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:03.298 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[21825]: from='osd.2 [v2:172.21.15.174:6818/77924542,v1:172.21.15.174:6819/77924542]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T22:04:03.298 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[21825]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T22:04:03.298 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[26730]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:04:03.298 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:03.298 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:03.298 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:03.298 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:03.298 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:03.298 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:03.298 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:03.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[26730]: Adjusting osd_memory_target on smithi174 to 5013M 2024-04-22T22:04:03.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:03.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:03.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[26730]: from='osd.2 [v2:172.21.15.174:6818/77924542,v1:172.21.15.174:6819/77924542]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T22:04:03.299 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:03 smithi174 ceph-mon[26730]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T22:04:03.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:03 smithi192 ceph-mon[22835]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:04:03.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:03 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:03.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:03 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:03.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:03 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:03.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:03 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:03.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:03 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:03.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:03 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:03.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:03 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:03.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:03 smithi192 ceph-mon[22835]: Adjusting osd_memory_target on smithi174 to 5013M 2024-04-22T22:04:03.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:03 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:03.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:03 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:03.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:03 smithi192 ceph-mon[22835]: from='osd.2 [v2:172.21.15.174:6818/77924542,v1:172.21.15.174:6819/77924542]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T22:04:03.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:03 smithi192 ceph-mon[22835]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-22T22:04:04.286 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:04 smithi174 ceph-mon[21825]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T22:04:04.286 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:04 smithi174 ceph-mon[21825]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T22:04:04.286 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:04 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:04:04.286 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:04 smithi174 ceph-mon[21825]: from='osd.2 [v2:172.21.15.174:6818/77924542,v1:172.21.15.174:6819/77924542]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:04:04.286 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:04 smithi174 ceph-mon[21825]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:04:04.287 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:04 smithi174 ceph-mon[26730]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T22:04:04.287 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:04 smithi174 ceph-mon[26730]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T22:04:04.287 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:04 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:04:04.287 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:04 smithi174 ceph-mon[26730]: from='osd.2 [v2:172.21.15.174:6818/77924542,v1:172.21.15.174:6819/77924542]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:04:04.287 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:04 smithi174 ceph-mon[26730]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:04:04.287 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:04:04 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2[35799]: 2024-04-22T22:04:04.175+0000 7fd2f7697700 -1 osd.2 0 waiting for initial osdmap 2024-04-22T22:04:04.287 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:04:04 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2[35799]: 2024-04-22T22:04:04.178+0000 7fd2f4032700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T22:04:04.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:04 smithi192 ceph-mon[22835]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-22T22:04:04.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:04 smithi192 ceph-mon[22835]: osdmap e16: 3 total, 2 up, 3 in 2024-04-22T22:04:04.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:04 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:04:04.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:04 smithi192 ceph-mon[22835]: from='osd.2 [v2:172.21.15.174:6818/77924542,v1:172.21.15.174:6819/77924542]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:04:04.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:04 smithi192 ceph-mon[22835]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:04:04.651 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T22:04:04.651 INFO:teuthology.orchestra.run.smithi174.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T22:04:04.651 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10+0 records in 2024-04-22T22:04:04.651 INFO:teuthology.orchestra.run.smithi174.stderr:10+0 records out 2024-04-22T22:04:04.651 INFO:teuthology.orchestra.run.smithi174.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125597 s, 835 MB/s 2024-04-22T22:04:04.651 INFO:teuthology.orchestra.run.smithi174.stderr:--> Zapping successful for: 2024-04-22T22:04:05.307 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi174:vg_nvme/lv_1 2024-04-22T22:04:05.560 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:05 smithi174 ceph-mon[21825]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:04:05.560 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:05 smithi174 ceph-mon[21825]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-22T22:04:05.560 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:05 smithi174 ceph-mon[21825]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T22:04:05.560 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:05 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:04:05.560 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:05 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:04:05.561 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:05 smithi174 ceph-mon[26730]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:04:05.561 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:05 smithi174 ceph-mon[26730]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-22T22:04:05.561 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:05 smithi174 ceph-mon[26730]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T22:04:05.561 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:05 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:04:05.561 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:05 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:04:05.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:05 smithi192 ceph-mon[22835]: pgmap v71: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:04:05.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:05 smithi192 ceph-mon[22835]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-22T22:04:05.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:05 smithi192 ceph-mon[22835]: osdmap e17: 3 total, 2 up, 3 in 2024-04-22T22:04:05.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:05 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:04:05.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:05 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:04:06.488 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:06 smithi174 ceph-mon[21825]: purged_snaps scrub starts 2024-04-22T22:04:06.488 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:06 smithi174 ceph-mon[21825]: purged_snaps scrub ok 2024-04-22T22:04:06.488 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:06 smithi174 ceph-mon[21825]: osd.2 [v2:172.21.15.174:6818/77924542,v1:172.21.15.174:6819/77924542] boot 2024-04-22T22:04:06.488 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:06 smithi174 ceph-mon[21825]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T22:04:06.488 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:06 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:04:06.488 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:06 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-22T22:04:06.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:06 smithi174 ceph-mon[26730]: purged_snaps scrub starts 2024-04-22T22:04:06.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:06 smithi174 ceph-mon[26730]: purged_snaps scrub ok 2024-04-22T22:04:06.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:06 smithi174 ceph-mon[26730]: osd.2 [v2:172.21.15.174:6818/77924542,v1:172.21.15.174:6819/77924542] boot 2024-04-22T22:04:06.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:06 smithi174 ceph-mon[26730]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T22:04:06.488 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:06 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:04:06.489 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:06 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-22T22:04:06.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:06 smithi192 ceph-mon[22835]: purged_snaps scrub starts 2024-04-22T22:04:06.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:06 smithi192 ceph-mon[22835]: purged_snaps scrub ok 2024-04-22T22:04:06.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:06 smithi192 ceph-mon[22835]: osd.2 [v2:172.21.15.174:6818/77924542,v1:172.21.15.174:6819/77924542] boot 2024-04-22T22:04:06.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:06 smithi192 ceph-mon[22835]: osdmap e18: 3 total, 3 up, 3 in 2024-04-22T22:04:06.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:06 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:04:06.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:06 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-22T22:04:07.518 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:07 smithi174 ceph-mon[26730]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:04:07.518 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:07 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-22T22:04:07.518 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:07 smithi174 ceph-mon[26730]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T22:04:07.518 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:07 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T22:04:07.518 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:07 smithi174 ceph-mon[21825]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:04:07.518 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:07 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-22T22:04:07.518 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:07 smithi174 ceph-mon[21825]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T22:04:07.519 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:07 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T22:04:07.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:07 smithi192 ceph-mon[22835]: pgmap v74: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-22T22:04:07.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:07 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-22T22:04:07.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:07 smithi192 ceph-mon[22835]: osdmap e19: 3 total, 3 up, 3 in 2024-04-22T22:04:07.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:07 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-22T22:04:07.772 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:04:07 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:04:07] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:04:08.315 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:04:08 smithi174 sudo[36911]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T22:04:08.315 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:04:08 smithi174 sudo[36911]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:04:08.315 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:04:08 smithi174 sudo[36911]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:04:08.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T22:04:08.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[21825]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T22:04:08.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:08.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:08.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:08.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:08.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:08.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:08.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T22:04:08.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[26730]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T22:04:08.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:08.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:08.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:08.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:08.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:08.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:08 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:08.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:08 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-22T22:04:08.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:08 smithi192 ceph-mon[22835]: osdmap e20: 3 total, 3 up, 3 in 2024-04-22T22:04:08.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:08 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:08.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:08 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:08.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:08 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:08.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:08 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:08.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:08 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:08.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:08 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:08.592 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:04:08 smithi174 sudo[36930]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T22:04:08.592 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:04:08 smithi174 sudo[36930]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:04:08.592 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:04:08 smithi174 sudo[36930]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:04:08.855 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:04:08 smithi174 sudo[36957]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T22:04:08.855 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:04:08 smithi174 sudo[36957]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:04:08.856 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:04:08 smithi174 sudo[36957]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:04:09.117 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:04:08 smithi174 sudo[37020]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T22:04:09.117 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:04:08 smithi174 sudo[37020]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:04:09.117 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:04:08 smithi174 sudo[37020]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:04:09.385 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:04:09 smithi174 sudo[37105]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T22:04:09.386 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:04:09 smithi174 sudo[37105]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:04:09.386 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:04:09 smithi174 sudo[37105]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:04:09.386 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[21825]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:09.386 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[21825]: Detected new or changed devices on smithi174 2024-04-22T22:04:09.386 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[21825]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:09.386 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[21825]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-22T22:04:09.386 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:09.386 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:04:09.386 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:04:09.387 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:09.387 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[21825]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T22:04:09.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[26730]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:09.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[26730]: Detected new or changed devices on smithi174 2024-04-22T22:04:09.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[26730]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:09.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[26730]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-22T22:04:09.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:09.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:04:09.388 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:04:09.388 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:09.388 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:09 smithi174 ceph-mon[26730]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T22:04:09.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:09 smithi192 ceph-mon[22835]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:09.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:09 smithi192 ceph-mon[22835]: Detected new or changed devices on smithi174 2024-04-22T22:04:09.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:09 smithi192 ceph-mon[22835]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:09.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:09 smithi192 ceph-mon[22835]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-22T22:04:09.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:09 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:09.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:09 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:04:09.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:09 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:04:09.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:09 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:09.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:09 smithi192 ceph-mon[22835]: osdmap e21: 3 total, 3 up, 3 in 2024-04-22T22:04:09.647 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 sudo[37132]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T22:04:09.647 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 sudo[37132]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:04:09.647 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:04:09 smithi174 sudo[37127]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T22:04:09.647 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:04:09 smithi174 sudo[37127]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:04:09.647 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:04:09 smithi174 sudo[37127]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:04:10.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 sudo[37132]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:04:10.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 sudo[37149]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T22:04:10.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 sudo[37149]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:04:10.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:09 smithi174 sudo[37149]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:04:10.397 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 sudo[37153]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T22:04:10.397 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 sudo[37153]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:04:10.397 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 sudo[37153]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:04:10.673 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 sudo[37158]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T22:04:10.673 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 sudo[37158]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:04:10.673 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 sudo[37158]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:04:10.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 sudo[25349]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-22T22:04:10.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 sudo[25349]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:04:10.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 sudo[25349]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:04:10.939 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[21825]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:04:10.939 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[21825]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T22:04:10.939 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[21825]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T22:04:10.939 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:04:10.939 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:04:10.939 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:04:10.939 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:04:10.939 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:04:10.939 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:04:10.940 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[26730]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:04:10.940 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[26730]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T22:04:10.940 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[26730]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T22:04:10.940 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:04:10.940 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:04:10.940 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:04:10.940 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:04:10.940 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:04:10.940 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:04:11.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 sudo[25352]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-22T22:04:11.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 sudo[25352]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-22T22:04:11.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 sudo[25352]: pam_unix(sudo:session): session closed for user root 2024-04-22T22:04:11.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 ceph-mon[22835]: from='client.24208 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi174:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:04:11.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 ceph-mon[22835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T22:04:11.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 ceph-mon[22835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T22:04:11.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:04:11.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:04:11.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:04:11.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:04:11.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:04:11.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:04:11.815 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:04:11] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:04:12.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:12.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T22:04:12.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T22:04:12.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T22:04:12.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:04:12.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:04:12.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:04:12.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T22:04:12.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-22T22:04:12.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:12.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:12.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:12.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:12.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:12.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:12.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:12.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2604853924' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0ce25dc-969a-4435-ab08-96d78791c719"}]: dispatch 2024-04-22T22:04:12.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2604853924' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0ce25dc-969a-4435-ab08-96d78791c719"}]': finished 2024-04-22T22:04:12.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T22:04:12.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:11 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:12.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:12.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T22:04:12.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T22:04:12.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T22:04:12.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:04:12.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:04:12.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:04:12.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T22:04:12.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-22T22:04:12.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:12.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:12.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:12.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:12.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:12.319 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:12.319 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:12.319 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2604853924' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0ce25dc-969a-4435-ab08-96d78791c719"}]: dispatch 2024-04-22T22:04:12.319 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2604853924' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0ce25dc-969a-4435-ab08-96d78791c719"}]': finished 2024-04-22T22:04:12.319 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T22:04:12.319 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:12.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: pgmap v80: 1 pgs: 1 active+clean; 449 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:12.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T22:04:12.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T22:04:12.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-22T22:04:12.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:04:12.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2604853924' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0ce25dc-969a-4435-ab08-96d78791c719"}]: dispatch 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2604853924' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0ce25dc-969a-4435-ab08-96d78791c719"}]': finished 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: osdmap e22: 4 total, 3 up, 4 in 2024-04-22T22:04:12.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:11 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:13.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:13 smithi192 ceph-mon[22835]: pgmap v81: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:13.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:13 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2320913631' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:04:13.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:13 smithi192 ceph-mon[22835]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-22T22:04:13.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:13 smithi192 ceph-mon[22835]: Cluster is now healthy 2024-04-22T22:04:13.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:13 smithi174 ceph-mon[21825]: pgmap v81: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:13.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:13 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2320913631' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:04:13.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:13 smithi174 ceph-mon[21825]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-22T22:04:13.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:13 smithi174 ceph-mon[21825]: Cluster is now healthy 2024-04-22T22:04:13.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:13 smithi174 ceph-mon[26730]: pgmap v81: 1 pgs: 1 active+clean; 449 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:13.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:13 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2320913631' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:04:13.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:13 smithi174 ceph-mon[26730]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-22T22:04:13.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:13 smithi174 ceph-mon[26730]: Cluster is now healthy 2024-04-22T22:04:14.815 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:04:14 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:04:14] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:04:15.173 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:15 smithi174 ceph-mon[21825]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 265 KiB/s wr, 18 op/s 2024-04-22T22:04:15.173 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:15 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:15.174 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:15 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:15.174 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:15 smithi174 ceph-mon[26730]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 265 KiB/s wr, 18 op/s 2024-04-22T22:04:15.174 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:15 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:15.174 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:15 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:15.248 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:15 smithi192 ceph-mon[22835]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 265 KiB/s wr, 18 op/s 2024-04-22T22:04:15.248 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:15 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:15.248 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:15 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:17.149 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:17 smithi192 ceph-mon[22835]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 189 KiB/s wr, 12 op/s 2024-04-22T22:04:17.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:17 smithi174 ceph-mon[21825]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 189 KiB/s wr, 12 op/s 2024-04-22T22:04:17.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:17 smithi174 ceph-mon[26730]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 189 KiB/s wr, 12 op/s 2024-04-22T22:04:19.257 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:19 smithi174 ceph-mon[21825]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 146 KiB/s wr, 10 op/s 2024-04-22T22:04:19.258 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:19 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T22:04:19.258 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:19 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:19.258 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:19 smithi174 ceph-mon[21825]: Deploying daemon osd.3 on smithi174 2024-04-22T22:04:19.258 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:19 smithi174 ceph-mon[26730]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 146 KiB/s wr, 10 op/s 2024-04-22T22:04:19.258 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:19 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T22:04:19.258 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:19 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:19.258 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:19 smithi174 ceph-mon[26730]: Deploying daemon osd.3 on smithi174 2024-04-22T22:04:19.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:19 smithi192 ceph-mon[22835]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 146 KiB/s wr, 10 op/s 2024-04-22T22:04:19.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:19 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-22T22:04:19.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:19 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:19.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:19 smithi192 ceph-mon[22835]: Deploying daemon osd.3 on smithi174 2024-04-22T22:04:21.065 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:21 smithi174 ceph-mon[26730]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 132 KiB/s wr, 9 op/s 2024-04-22T22:04:21.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:21 smithi174 ceph-mon[21825]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 132 KiB/s wr, 9 op/s 2024-04-22T22:04:21.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:21 smithi192 ceph-mon[22835]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 132 KiB/s wr, 9 op/s 2024-04-22T22:04:23.213 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:23 smithi174 ceph-mon[26730]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-22T22:04:23.213 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:23 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:23.213 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:23 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:23.213 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:23 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:23.213 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:23 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:23.213 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:23 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:23.213 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:23 smithi174 ceph-mon[21825]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-22T22:04:23.213 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:23.213 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:23.213 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:23.214 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:23.214 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:23.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:23 smithi192 ceph-mon[22835]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-04-22T22:04:23.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:23 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:23.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:23 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:23.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:23 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:23.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:23 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:23.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:23 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:25.137 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:25 smithi174 ceph-mon[21825]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 110 KiB/s wr, 7 op/s 2024-04-22T22:04:25.137 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:25 smithi174 ceph-mon[26730]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 110 KiB/s wr, 7 op/s 2024-04-22T22:04:25.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:25 smithi192 ceph-mon[22835]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.3 KiB/s rd, 110 KiB/s wr, 7 op/s 2024-04-22T22:04:26.119 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:26 smithi174 ceph-mon[21825]: from='osd.3 [v2:172.21.15.174:6826/1588342953,v1:172.21.15.174:6827/1588342953]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T22:04:26.119 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:26 smithi174 ceph-mon[21825]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T22:04:26.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:26 smithi174 ceph-mon[26730]: from='osd.3 [v2:172.21.15.174:6826/1588342953,v1:172.21.15.174:6827/1588342953]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T22:04:26.119 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:26 smithi174 ceph-mon[26730]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T22:04:26.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:26 smithi192 ceph-mon[22835]: from='osd.3 [v2:172.21.15.174:6826/1588342953,v1:172.21.15.174:6827/1588342953]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T22:04:26.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:26 smithi192 ceph-mon[22835]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-22T22:04:26.457 INFO:teuthology.orchestra.run.smithi174.stdout:Created osd(s) 3 on host 'smithi174' 2024-04-22T22:04:27.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:27.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T22:04:27.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T22:04:27.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:27.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='osd.3 [v2:172.21.15.174:6826/1588342953,v1:172.21.15.174:6827/1588342953]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:04:27.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:04:27.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:27.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:27.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:27.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:27.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:27.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:27.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:27.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:27.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: Adjusting osd_memory_target on smithi174 to 3760M 2024-04-22T22:04:27.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:27.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:27.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:27.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T22:04:27.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T22:04:27.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:27.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='osd.3 [v2:172.21.15.174:6826/1588342953,v1:172.21.15.174:6827/1588342953]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:04:27.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:04:27.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:27.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:27.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:27.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:27.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:27.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:27.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:27.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:27.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: Adjusting osd_memory_target on smithi174 to 3760M 2024-04-22T22:04:27.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:27.318 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:27.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: osdmap e23: 4 total, 3 up, 4 in 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='osd.3 [v2:172.21.15.174:6826/1588342953,v1:172.21.15.174:6827/1588342953]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]: dispatch 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:27.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: Adjusting osd_memory_target on smithi174 to 3760M 2024-04-22T22:04:27.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:27.320 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:27.690 DEBUG:teuthology.orchestra.run.smithi174:osd.3> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.3.service 2024-04-22T22:04:27.693 INFO:tasks.cephadm:Deploying osd.4 on smithi192 with /dev/vg_nvme/lv_4... 2024-04-22T22:04:27.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-22T22:04:28.213 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:28 smithi174 ceph-mon[21825]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-22T22:04:28.213 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:28 smithi174 ceph-mon[21825]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T22:04:28.213 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:28 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:28.213 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:28 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:28.214 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:28 smithi174 ceph-mon[26730]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-22T22:04:28.214 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:28 smithi174 ceph-mon[26730]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T22:04:28.214 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:28 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:28.214 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:28 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:28.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:28 smithi192 ceph-mon[22835]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi174", "root=default"]}]': finished 2024-04-22T22:04:28.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:28 smithi192 ceph-mon[22835]: osdmap e24: 4 total, 3 up, 4 in 2024-04-22T22:04:28.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:28 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:28.285 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:28 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:29.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:29 smithi174 ceph-mon[21825]: purged_snaps scrub starts 2024-04-22T22:04:29.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:29 smithi174 ceph-mon[21825]: purged_snaps scrub ok 2024-04-22T22:04:29.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:29 smithi174 ceph-mon[21825]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:29.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:29 smithi174 ceph-mon[21825]: osd.3 [v2:172.21.15.174:6826/1588342953,v1:172.21.15.174:6827/1588342953] boot 2024-04-22T22:04:29.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:29 smithi174 ceph-mon[21825]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T22:04:29.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:29 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:29.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:29 smithi174 ceph-mon[26730]: purged_snaps scrub starts 2024-04-22T22:04:29.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:29 smithi174 ceph-mon[26730]: purged_snaps scrub ok 2024-04-22T22:04:29.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:29 smithi174 ceph-mon[26730]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:29.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:29 smithi174 ceph-mon[26730]: osd.3 [v2:172.21.15.174:6826/1588342953,v1:172.21.15.174:6827/1588342953] boot 2024-04-22T22:04:29.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:29 smithi174 ceph-mon[26730]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T22:04:29.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:29 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:29 smithi192 ceph-mon[22835]: purged_snaps scrub starts 2024-04-22T22:04:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:29 smithi192 ceph-mon[22835]: purged_snaps scrub ok 2024-04-22T22:04:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:29 smithi192 ceph-mon[22835]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-22T22:04:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:29 smithi192 ceph-mon[22835]: osd.3 [v2:172.21.15.174:6826/1588342953,v1:172.21.15.174:6827/1588342953] boot 2024-04-22T22:04:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:29 smithi192 ceph-mon[22835]: osdmap e25: 4 total, 4 up, 4 in 2024-04-22T22:04:29.156 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:29 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:04:29.629 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-22T22:04:29.629 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-22T22:04:29.629 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-04-22T22:04:29.629 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-04-22T22:04:29.629 INFO:teuthology.orchestra.run.smithi192.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123647 s, 848 MB/s 2024-04-22T22:04:29.629 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-04-22T22:04:30.065 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:04:29 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:04:29] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:04:30.174 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_4 2024-04-22T22:04:31.024 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:30 smithi192 ceph-mon[22835]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:31.024 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:30 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:31.024 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:30 smithi192 ceph-mon[22835]: Detected new or changed devices on smithi174 2024-04-22T22:04:31.024 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:30 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:31.024 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:30 smithi192 ceph-mon[22835]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T22:04:31.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:30 smithi174 ceph-mon[21825]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:31.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:30 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:31.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:30 smithi174 ceph-mon[21825]: Detected new or changed devices on smithi174 2024-04-22T22:04:31.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:30 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:31.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:30 smithi174 ceph-mon[21825]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T22:04:31.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:30 smithi174 ceph-mon[26730]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:31.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:30 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:31.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:30 smithi174 ceph-mon[26730]: Detected new or changed devices on smithi174 2024-04-22T22:04:31.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:30 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:31.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:30 smithi174 ceph-mon[26730]: osdmap e26: 4 total, 4 up, 4 in 2024-04-22T22:04:32.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:32 smithi192 ceph-mon[22835]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:32.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:32 smithi192 ceph-mon[22835]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:04:32.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:32 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:04:32.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:32 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:04:32.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:32 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:33.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:32 smithi174 ceph-mon[21825]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:33.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:32 smithi174 ceph-mon[21825]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:04:33.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:32 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:04:33.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:32 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:04:33.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:32 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:33.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:32 smithi174 ceph-mon[26730]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:33.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:32 smithi174 ceph-mon[26730]: from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:04:33.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:32 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:04:33.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:32 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:04:33.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:32 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:35.065 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:04:34] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:04:35.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[21825]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:35.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[21825]: from='client.? 172.21.15.192:0/1511163814' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b95a43b7-c458-4f35-8494-52af4cc55af8"}]: dispatch 2024-04-22T22:04:35.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[21825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b95a43b7-c458-4f35-8494-52af4cc55af8"}]: dispatch 2024-04-22T22:04:35.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[21825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b95a43b7-c458-4f35-8494-52af4cc55af8"}]': finished 2024-04-22T22:04:35.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[21825]: osdmap e27: 5 total, 4 up, 5 in 2024-04-22T22:04:35.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:35.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[21825]: from='client.? 172.21.15.192:0/2840762151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:04:35.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:35.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:35.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[26730]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:35.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[26730]: from='client.? 172.21.15.192:0/1511163814' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b95a43b7-c458-4f35-8494-52af4cc55af8"}]: dispatch 2024-04-22T22:04:35.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[26730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b95a43b7-c458-4f35-8494-52af4cc55af8"}]: dispatch 2024-04-22T22:04:35.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[26730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b95a43b7-c458-4f35-8494-52af4cc55af8"}]': finished 2024-04-22T22:04:35.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[26730]: osdmap e27: 5 total, 4 up, 5 in 2024-04-22T22:04:35.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:35.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[26730]: from='client.? 172.21.15.192:0/2840762151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:04:35.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:35.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:34 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:35.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:34 smithi192 ceph-mon[22835]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:35.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:34 smithi192 ceph-mon[22835]: from='client.? 172.21.15.192:0/1511163814' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b95a43b7-c458-4f35-8494-52af4cc55af8"}]: dispatch 2024-04-22T22:04:35.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:34 smithi192 ceph-mon[22835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b95a43b7-c458-4f35-8494-52af4cc55af8"}]: dispatch 2024-04-22T22:04:35.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:34 smithi192 ceph-mon[22835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b95a43b7-c458-4f35-8494-52af4cc55af8"}]': finished 2024-04-22T22:04:35.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:34 smithi192 ceph-mon[22835]: osdmap e27: 5 total, 4 up, 5 in 2024-04-22T22:04:35.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:34 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:35.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:34 smithi192 ceph-mon[22835]: from='client.? 172.21.15.192:0/2840762151' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:04:35.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:34 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:35.068 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:34 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:36.800 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:36 smithi192 ceph-mon[22835]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:37.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:36 smithi174 ceph-mon[21825]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:37.065 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:36 smithi174 ceph-mon[26730]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:39.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:39 smithi174 ceph-mon[21825]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:39.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:39 smithi174 ceph-mon[26730]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:39.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:39 smithi192 ceph-mon[22835]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:41.311 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:41 smithi192 ceph-mon[22835]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:41.312 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:41 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T22:04:41.312 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:41 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:41.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:41 smithi174 ceph-mon[21825]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:41.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T22:04:41.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:41 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:41.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:41 smithi174 ceph-mon[26730]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:41.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-22T22:04:41.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:41 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:42.181 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:42 smithi192 ceph-mon[22835]: Deploying daemon osd.4 on smithi192 2024-04-22T22:04:42.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:42 smithi174 ceph-mon[21825]: Deploying daemon osd.4 on smithi192 2024-04-22T22:04:42.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:42 smithi174 ceph-mon[26730]: Deploying daemon osd.4 on smithi192 2024-04-22T22:04:43.308 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:43 smithi192 ceph-mon[22835]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:43.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:43 smithi174 ceph-mon[21825]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:43.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:43 smithi174 ceph-mon[26730]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:45.245 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:45 smithi192 ceph-mon[22835]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:45.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:45 smithi174 ceph-mon[21825]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:45.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:45 smithi174 ceph-mon[26730]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:46.560 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:46 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:46.561 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:46 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:46.561 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:46 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:46.561 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:46 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:46.561 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:46 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:46.561 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:46 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:46.561 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:46 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:46.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:04:46] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:04:46.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:46.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:46.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:46.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:46.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:46.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:46.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:46.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:46.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:46.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:46.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:46.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:46.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:46.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:46 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:47.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:47 smithi192 ceph-mon[22835]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:47.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:47 smithi174 ceph-mon[21825]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:47.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:47 smithi174 ceph-mon[26730]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:47.704 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 4 on host 'smithi192' 2024-04-22T22:04:48.239 DEBUG:teuthology.orchestra.run.smithi192:osd.4> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.4.service 2024-04-22T22:04:48.241 INFO:tasks.cephadm:Deploying osd.5 on smithi192 with /dev/vg_nvme/lv_3... 2024-04-22T22:04:48.242 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-22T22:04:48.566 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:04:48 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:04:48.418+0000 7f59f14c43c0 -1 osd.4 0 log_to_monitors true 2024-04-22T22:04:48.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:48 smithi192 ceph-mon[22835]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:48.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:48 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:48.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:48 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:48.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:48 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:48.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:48 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:48.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:48 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:48.887 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:48 smithi192 ceph-mon[22835]: Adjusting osd_memory_target on smithi192 to 16064M 2024-04-22T22:04:48.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:48 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:48.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:48 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:48.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:48 smithi192 ceph-mon[22835]: from='osd.4 [v2:172.21.15.192:6800/469106314,v1:172.21.15.192:6801/469106314]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T22:04:48.888 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:48 smithi192 ceph-mon[22835]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T22:04:49.065 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[26730]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:49.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:49.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:49.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:49.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:49.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:49.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[26730]: Adjusting osd_memory_target on smithi192 to 16064M 2024-04-22T22:04:49.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:49.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:49.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[26730]: from='osd.4 [v2:172.21.15.192:6800/469106314,v1:172.21.15.192:6801/469106314]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T22:04:49.067 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[26730]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T22:04:49.067 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[21825]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:49.067 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:49.067 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:49.067 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:49.067 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:49.068 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:04:49.068 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[21825]: Adjusting osd_memory_target on smithi192 to 16064M 2024-04-22T22:04:49.068 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:49.068 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:49.068 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[21825]: from='osd.4 [v2:172.21.15.192:6800/469106314,v1:172.21.15.192:6801/469106314]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T22:04:49.068 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:48 smithi174 ceph-mon[21825]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-22T22:04:50.013 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-22T22:04:50.013 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-22T22:04:50.013 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-04-22T22:04:50.013 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-04-22T22:04:50.013 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140199 s, 748 MB/s 2024-04-22T22:04:50.014 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-04-22T22:04:50.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:50 smithi174 ceph-mon[26730]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T22:04:50.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:50 smithi174 ceph-mon[26730]: osdmap e28: 5 total, 4 up, 5 in 2024-04-22T22:04:50.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:50 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:50.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:50 smithi174 ceph-mon[26730]: from='osd.4 [v2:172.21.15.192:6800/469106314,v1:172.21.15.192:6801/469106314]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:04:50.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:50 smithi174 ceph-mon[26730]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:04:50.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:50 smithi174 ceph-mon[21825]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T22:04:50.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:50 smithi174 ceph-mon[21825]: osdmap e28: 5 total, 4 up, 5 in 2024-04-22T22:04:50.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:50 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:50.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:50 smithi174 ceph-mon[21825]: from='osd.4 [v2:172.21.15.192:6800/469106314,v1:172.21.15.192:6801/469106314]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:04:50.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:50 smithi174 ceph-mon[21825]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:04:50.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:50 smithi192 ceph-mon[22835]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-22T22:04:50.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:50 smithi192 ceph-mon[22835]: osdmap e28: 5 total, 4 up, 5 in 2024-04-22T22:04:50.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:50 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:50.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:50 smithi192 ceph-mon[22835]: from='osd.4 [v2:172.21.15.192:6800/469106314,v1:172.21.15.192:6801/469106314]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:04:50.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:50 smithi192 ceph-mon[22835]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:04:50.567 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:04:50 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:04:50.337+0000 7f59e7ec7700 -1 osd.4 0 waiting for initial osdmap 2024-04-22T22:04:50.567 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:04:50 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:04:50.341+0000 7f59e205d700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T22:04:50.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_3 2024-04-22T22:04:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:51 smithi192 ceph-mon[22835]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:51 smithi192 ceph-mon[22835]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-22T22:04:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:51 smithi192 ceph-mon[22835]: osdmap e29: 5 total, 4 up, 5 in 2024-04-22T22:04:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:51 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:51 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:51.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:51 smithi174 ceph-mon[26730]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:51.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:51 smithi174 ceph-mon[26730]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-22T22:04:51.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:51 smithi174 ceph-mon[26730]: osdmap e29: 5 total, 4 up, 5 in 2024-04-22T22:04:51.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:51 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:51.702 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:51 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:51.702 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:51 smithi174 ceph-mon[21825]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:51.703 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:51 smithi174 ceph-mon[21825]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-22T22:04:51.703 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:51 smithi174 ceph-mon[21825]: osdmap e29: 5 total, 4 up, 5 in 2024-04-22T22:04:51.703 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:51 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:51.703 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:51 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:52.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:52 smithi192 ceph-mon[22835]: purged_snaps scrub starts 2024-04-22T22:04:52.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:52 smithi192 ceph-mon[22835]: purged_snaps scrub ok 2024-04-22T22:04:52.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:52 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:52.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:52 smithi192 ceph-mon[22835]: osd.4 [v2:172.21.15.192:6800/469106314,v1:172.21.15.192:6801/469106314] boot 2024-04-22T22:04:52.415 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:52 smithi192 ceph-mon[22835]: osdmap e30: 5 total, 5 up, 5 in 2024-04-22T22:04:52.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:52 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:52.416 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:52 smithi192 ceph-mon[22835]: osdmap e31: 5 total, 5 up, 5 in 2024-04-22T22:04:52.624 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[21825]: purged_snaps scrub starts 2024-04-22T22:04:52.624 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[21825]: purged_snaps scrub ok 2024-04-22T22:04:52.624 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:52.624 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[21825]: osd.4 [v2:172.21.15.192:6800/469106314,v1:172.21.15.192:6801/469106314] boot 2024-04-22T22:04:52.624 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[21825]: osdmap e30: 5 total, 5 up, 5 in 2024-04-22T22:04:52.624 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:52.624 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[21825]: osdmap e31: 5 total, 5 up, 5 in 2024-04-22T22:04:52.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[26730]: purged_snaps scrub starts 2024-04-22T22:04:52.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[26730]: purged_snaps scrub ok 2024-04-22T22:04:52.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:52.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[26730]: osd.4 [v2:172.21.15.192:6800/469106314,v1:172.21.15.192:6801/469106314] boot 2024-04-22T22:04:52.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[26730]: osdmap e30: 5 total, 5 up, 5 in 2024-04-22T22:04:52.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:04:52.625 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:52 smithi174 ceph-mon[26730]: osdmap e31: 5 total, 5 up, 5 in 2024-04-22T22:04:53.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:53 smithi192 ceph-mon[22835]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:53.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:53 smithi192 ceph-mon[22835]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:04:53.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:53 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:04:53.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:53 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:04:53.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:53 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:53.668 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:53 smithi192 ceph-mon[22835]: osdmap e32: 5 total, 5 up, 5 in 2024-04-22T22:04:53.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:53 smithi174 ceph-mon[26730]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:53.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:53 smithi174 ceph-mon[26730]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:04:53.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:53 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:04:53.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:53 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:04:53.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:53 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:53.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:53 smithi174 ceph-mon[26730]: osdmap e32: 5 total, 5 up, 5 in 2024-04-22T22:04:53.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:53 smithi174 ceph-mon[21825]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-22T22:04:53.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:53 smithi174 ceph-mon[21825]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:04:53.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:53 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:04:53.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:53 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:04:53.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:53 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:53.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:53 smithi174 ceph-mon[21825]: osdmap e32: 5 total, 5 up, 5 in 2024-04-22T22:04:54.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:04:54.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:04:54.681 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:54 smithi192 ceph-mon[22835]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T22:04:54.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:04:54.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:04:54.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:54 smithi174 ceph-mon[21825]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T22:04:54.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:04:54.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:04:54.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:54 smithi174 ceph-mon[26730]: osdmap e33: 5 total, 5 up, 5 in 2024-04-22T22:04:55.230 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:04:54 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:04:54] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:04:55.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:04:55] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:04:55.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: pgmap v113: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:04:55.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-22T22:04:55.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='client.? 172.21.15.192:0/886354093' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3400fe78-1500-44a8-a526-2a80969d5c54"}]: dispatch 2024-04-22T22:04:55.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3400fe78-1500-44a8-a526-2a80969d5c54"}]: dispatch 2024-04-22T22:04:55.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3400fe78-1500-44a8-a526-2a80969d5c54"}]': finished 2024-04-22T22:04:55.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: osdmap e34: 6 total, 5 up, 6 in 2024-04-22T22:04:55.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:04:55.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:55.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='client.? 172.21.15.192:0/445559455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: pgmap v113: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='client.? 172.21.15.192:0/886354093' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3400fe78-1500-44a8-a526-2a80969d5c54"}]: dispatch 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3400fe78-1500-44a8-a526-2a80969d5c54"}]: dispatch 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3400fe78-1500-44a8-a526-2a80969d5c54"}]': finished 2024-04-22T22:04:55.567 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: osdmap e34: 6 total, 5 up, 6 in 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='client.? 172.21.15.192:0/445559455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:55.568 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.569 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: pgmap v113: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:04:55.569 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-22T22:04:55.569 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='client.? 172.21.15.192:0/886354093' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3400fe78-1500-44a8-a526-2a80969d5c54"}]: dispatch 2024-04-22T22:04:55.569 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3400fe78-1500-44a8-a526-2a80969d5c54"}]: dispatch 2024-04-22T22:04:55.569 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3400fe78-1500-44a8-a526-2a80969d5c54"}]': finished 2024-04-22T22:04:55.569 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: osdmap e34: 6 total, 5 up, 6 in 2024-04-22T22:04:55.569 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:04:55.569 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.569 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.569 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:55.569 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:55.569 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:55.569 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.570 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.570 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:55.570 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='client.? 172.21.15.192:0/445559455' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:04:55.570 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:04:55.570 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:04:55.570 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:04:55.570 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:04:56.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:56 smithi192 ceph-mon[22835]: Detected new or changed devices on smithi192 2024-04-22T22:04:56.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:56 smithi192 ceph-mon[22835]: pgmap v116: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:04:56.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:56 smithi192 ceph-mon[22835]: pgmap v117: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:04:56.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:56 smithi174 ceph-mon[21825]: Detected new or changed devices on smithi192 2024-04-22T22:04:56.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:56 smithi174 ceph-mon[21825]: pgmap v116: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:04:56.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:56 smithi174 ceph-mon[21825]: pgmap v117: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:04:56.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:56 smithi174 ceph-mon[26730]: Detected new or changed devices on smithi192 2024-04-22T22:04:56.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:56 smithi174 ceph-mon[26730]: pgmap v116: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:04:56.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:56 smithi174 ceph-mon[26730]: pgmap v117: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:04:58.294 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:58 smithi192 ceph-mon[22835]: pgmap v118: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:04:58.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:58 smithi174 ceph-mon[21825]: pgmap v118: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:04:58.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:58 smithi174 ceph-mon[26730]: pgmap v118: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:04:59.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:59 smithi192 ceph-mon[22835]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-22T22:04:59.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:04:59 smithi192 ceph-mon[22835]: Cluster is now healthy 2024-04-22T22:04:59.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:59 smithi174 ceph-mon[21825]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-22T22:04:59.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:04:59 smithi174 ceph-mon[21825]: Cluster is now healthy 2024-04-22T22:04:59.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:59 smithi174 ceph-mon[26730]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-22T22:04:59.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:04:59 smithi174 ceph-mon[26730]: Cluster is now healthy 2024-04-22T22:05:00.486 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:00 smithi192 ceph-mon[22835]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-22T22:05:00.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:00 smithi174 ceph-mon[21825]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-22T22:05:00.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:00 smithi174 ceph-mon[26730]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-22T22:05:02.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:02 smithi174 ceph-mon[21825]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-22T22:05:02.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:02 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T22:05:02.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:02 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:02.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:02 smithi174 ceph-mon[26730]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-22T22:05:02.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:02 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T22:05:02.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:02 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:02.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:02 smithi192 ceph-mon[22835]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-04-22T22:05:02.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:02 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-22T22:05:02.629 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:02 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:03.543 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:03 smithi192 ceph-mon[22835]: Deploying daemon osd.5 on smithi192 2024-04-22T22:05:03.616 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:03 smithi174 ceph-mon[21825]: Deploying daemon osd.5 on smithi192 2024-04-22T22:05:03.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:03 smithi174 ceph-mon[26730]: Deploying daemon osd.5 on smithi192 2024-04-22T22:05:04.463 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:04 smithi174 ceph-mon[21825]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-22T22:05:04.464 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:04 smithi174 ceph-mon[26730]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-22T22:05:04.622 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:04 smithi192 ceph-mon[22835]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-22T22:05:06.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:06 smithi174 ceph-mon[21825]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-22T22:05:06.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:06 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:06.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:06 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:06.566 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:05:06 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:05:06] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:05:06.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:06 smithi174 ceph-mon[26730]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-22T22:05:06.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:06 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:06.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:06 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:06.581 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:06 smithi192 ceph-mon[22835]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-04-22T22:05:06.581 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:06 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:06.581 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:06 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:07.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:07 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:07.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:07 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:07.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:07 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:07.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:07 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:07.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:07 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:07.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:07 smithi192 ceph-mon[22835]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-22T22:05:08.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:07 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:08.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:07 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:08.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:07 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:08.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:07 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:08.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:07 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:08.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:07 smithi174 ceph-mon[21825]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-22T22:05:08.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:07 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:08.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:07 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:08.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:07 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:08.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:07 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:08.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:07 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:08.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:07 smithi174 ceph-mon[26730]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-22T22:05:09.041 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 5 on host 'smithi192' 2024-04-22T22:05:09.569 DEBUG:teuthology.orchestra.run.smithi192:osd.5> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.5.service 2024-04-22T22:05:09.571 INFO:tasks.cephadm:Deploying osd.6 on smithi192 with /dev/vg_nvme/lv_2... 2024-04-22T22:05:09.571 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-22T22:05:09.914 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:05:09 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:05:09.776+0000 7f31ad86f3c0 -1 osd.5 0 log_to_monitors true 2024-04-22T22:05:10.208 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:10.208 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:10.208 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:10.208 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:10.209 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:10.209 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:10.209 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:10 smithi192 ceph-mon[22835]: Adjusting osd_memory_target on smithi192 to 8032M 2024-04-22T22:05:10.209 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:10.209 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:10 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:10.209 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:10 smithi192 ceph-mon[22835]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-22T22:05:10.209 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:10 smithi192 ceph-mon[22835]: from='osd.5 [v2:172.21.15.192:6808/645383067,v1:172.21.15.192:6809/645383067]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T22:05:10.209 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:10 smithi192 ceph-mon[22835]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T22:05:10.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:10.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:10.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:10.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:10.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:10.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:10.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[21825]: Adjusting osd_memory_target on smithi192 to 8032M 2024-04-22T22:05:10.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:10.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:10.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[21825]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-22T22:05:10.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[21825]: from='osd.5 [v2:172.21.15.192:6808/645383067,v1:172.21.15.192:6809/645383067]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T22:05:10.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[21825]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T22:05:10.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:10.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:10.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:10.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:10.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:10.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:10.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[26730]: Adjusting osd_memory_target on smithi192 to 8032M 2024-04-22T22:05:10.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:10.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:10.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[26730]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-22T22:05:10.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[26730]: from='osd.5 [v2:172.21.15.192:6808/645383067,v1:172.21.15.192:6809/645383067]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T22:05:10.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:10 smithi174 ceph-mon[26730]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-22T22:05:11.545 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:05:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:05:11.128+0000 7f31a4272700 -1 osd.5 0 waiting for initial osdmap 2024-04-22T22:05:11.545 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:05:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:05:11.133+0000 7f319dc07700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T22:05:11.546 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:11 smithi192 ceph-mon[22835]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T22:05:11.546 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:11 smithi192 ceph-mon[22835]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T22:05:11.546 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:11 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:05:11.546 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:11 smithi192 ceph-mon[22835]: from='osd.5 [v2:172.21.15.192:6808/645383067,v1:172.21.15.192:6809/645383067]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:11.546 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:11 smithi192 ceph-mon[22835]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:11.546 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-22T22:05:11.546 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-22T22:05:11.546 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-04-22T22:05:11.547 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-04-22T22:05:11.547 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0142469 s, 736 MB/s 2024-04-22T22:05:11.547 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-04-22T22:05:11.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:11 smithi174 ceph-mon[21825]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T22:05:11.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:11 smithi174 ceph-mon[21825]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T22:05:11.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:11 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:05:11.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:11 smithi174 ceph-mon[21825]: from='osd.5 [v2:172.21.15.192:6808/645383067,v1:172.21.15.192:6809/645383067]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:11.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:11 smithi174 ceph-mon[21825]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:11.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:11 smithi174 ceph-mon[26730]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-22T22:05:11.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:11 smithi174 ceph-mon[26730]: osdmap e35: 6 total, 5 up, 6 in 2024-04-22T22:05:11.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:11 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:05:11.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:11 smithi174 ceph-mon[26730]: from='osd.5 [v2:172.21.15.192:6808/645383067,v1:172.21.15.192:6809/645383067]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:11.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:11 smithi174 ceph-mon[26730]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:12.110 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_2 2024-04-22T22:05:12.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:12 smithi192 ceph-mon[22835]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-22T22:05:12.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:12 smithi192 ceph-mon[22835]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T22:05:12.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:12 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:05:12.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:12 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:05:12.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:12 smithi192 ceph-mon[22835]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:05:12.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:12 smithi192 ceph-mon[22835]: osd.5 [v2:172.21.15.192:6808/645383067,v1:172.21.15.192:6809/645383067] boot 2024-04-22T22:05:12.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:12 smithi192 ceph-mon[22835]: osdmap e37: 6 total, 6 up, 6 in 2024-04-22T22:05:12.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:12 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:05:12.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[21825]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-22T22:05:12.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[21825]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T22:05:12.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:05:12.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:05:12.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[21825]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:05:12.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[21825]: osd.5 [v2:172.21.15.192:6808/645383067,v1:172.21.15.192:6809/645383067] boot 2024-04-22T22:05:12.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[21825]: osdmap e37: 6 total, 6 up, 6 in 2024-04-22T22:05:12.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:05:12.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[26730]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-22T22:05:12.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[26730]: osdmap e36: 6 total, 5 up, 6 in 2024-04-22T22:05:12.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:05:12.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:05:12.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[26730]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-22T22:05:12.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[26730]: osd.5 [v2:172.21.15.192:6808/645383067,v1:172.21.15.192:6809/645383067] boot 2024-04-22T22:05:12.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[26730]: osdmap e37: 6 total, 6 up, 6 in 2024-04-22T22:05:12.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:12 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:05:13.299 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:13 smithi192 ceph-mon[22835]: purged_snaps scrub starts 2024-04-22T22:05:13.299 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:13 smithi192 ceph-mon[22835]: purged_snaps scrub ok 2024-04-22T22:05:13.299 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:13 smithi192 ceph-mon[22835]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T22:05:13.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:13 smithi174 ceph-mon[21825]: purged_snaps scrub starts 2024-04-22T22:05:13.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:13 smithi174 ceph-mon[21825]: purged_snaps scrub ok 2024-04-22T22:05:13.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:13 smithi174 ceph-mon[21825]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T22:05:13.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:13 smithi174 ceph-mon[26730]: purged_snaps scrub starts 2024-04-22T22:05:13.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:13 smithi174 ceph-mon[26730]: purged_snaps scrub ok 2024-04-22T22:05:13.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:13 smithi174 ceph-mon[26730]: osdmap e38: 6 total, 6 up, 6 in 2024-04-22T22:05:14.371 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:14 smithi192 ceph-mon[22835]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:05:14.371 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:14 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:05:14.371 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:14 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:05:14.371 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:14 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:14.372 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:14 smithi192 ceph-mon[22835]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T22:05:14.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:14 smithi174 ceph-mon[21825]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:05:14.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:05:14.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:05:14.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:14 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:14.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:14 smithi174 ceph-mon[21825]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T22:05:14.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:14 smithi174 ceph-mon[26730]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:05:14.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:14 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:05:14.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:14 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:05:14.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:14 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:14.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:14 smithi174 ceph-mon[26730]: osdmap e39: 6 total, 6 up, 6 in 2024-04-22T22:05:15.393 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:15 smithi192 ceph-mon[22835]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:05:15.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:15 smithi174 ceph-mon[21825]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:05:15.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:15 smithi174 ceph-mon[26730]: from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:05:15.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:05:15 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:05:15] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:05:16.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T22:05:16.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: pgmap v133: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:05:16.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:16.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: Detected new or changed devices on smithi192 2024-04-22T22:05:16.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:16.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:16.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:16.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:16.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: pgmap v134: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:05:16.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:16.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: from='client.? 172.21.15.192:0/865391417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f446ce85-1bc3-4f2c-a013-7841180ee701"}]: dispatch 2024-04-22T22:05:16.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f446ce85-1bc3-4f2c-a013-7841180ee701"}]: dispatch 2024-04-22T22:05:16.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f446ce85-1bc3-4f2c-a013-7841180ee701"}]': finished 2024-04-22T22:05:16.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: osdmap e41: 7 total, 6 up, 7 in 2024-04-22T22:05:16.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:16 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:16.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T22:05:16.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: pgmap v133: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:05:16.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:16.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: Detected new or changed devices on smithi192 2024-04-22T22:05:16.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:16.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:16.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:16.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:16.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: pgmap v134: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:05:16.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:16.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: from='client.? 172.21.15.192:0/865391417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f446ce85-1bc3-4f2c-a013-7841180ee701"}]: dispatch 2024-04-22T22:05:16.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f446ce85-1bc3-4f2c-a013-7841180ee701"}]: dispatch 2024-04-22T22:05:16.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f446ce85-1bc3-4f2c-a013-7841180ee701"}]': finished 2024-04-22T22:05:16.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: osdmap e41: 7 total, 6 up, 7 in 2024-04-22T22:05:16.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:16.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: osdmap e40: 6 total, 6 up, 6 in 2024-04-22T22:05:16.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: pgmap v133: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:05:16.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:16.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: Detected new or changed devices on smithi192 2024-04-22T22:05:16.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:16.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:16.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:16.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:16.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: pgmap v134: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:05:16.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:16.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: from='client.? 172.21.15.192:0/865391417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f446ce85-1bc3-4f2c-a013-7841180ee701"}]: dispatch 2024-04-22T22:05:16.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f446ce85-1bc3-4f2c-a013-7841180ee701"}]: dispatch 2024-04-22T22:05:16.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f446ce85-1bc3-4f2c-a013-7841180ee701"}]': finished 2024-04-22T22:05:16.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: osdmap e41: 7 total, 6 up, 7 in 2024-04-22T22:05:16.567 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:17.065 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:05:16 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:05:16] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:05:17.488 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:17 smithi192 ceph-mon[22835]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-22T22:05:17.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:17 smithi192 ceph-mon[22835]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-22T22:05:17.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:17 smithi192 ceph-mon[22835]: from='client.? 172.21.15.192:0/1460918854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:05:17.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:17 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:17.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:17 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:17.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:17 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:17.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:17 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:17.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:17 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:17.489 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:17 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:17.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[21825]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-22T22:05:17.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[21825]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-22T22:05:17.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[21825]: from='client.? 172.21.15.192:0/1460918854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:05:17.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:17.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:17.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:17.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:17.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:17.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:17.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[26730]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-22T22:05:17.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[26730]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-22T22:05:17.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[26730]: from='client.? 172.21.15.192:0/1460918854' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:05:17.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:17.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:17.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:17.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:17.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:17.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:17 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:18.511 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:18 smithi192 ceph-mon[22835]: pgmap v136: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:05:18.512 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:18 smithi192 ceph-mon[22835]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-22T22:05:18.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:18 smithi174 ceph-mon[26730]: pgmap v136: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:05:18.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:18 smithi174 ceph-mon[26730]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-22T22:05:18.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:18 smithi174 ceph-mon[21825]: pgmap v136: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-22T22:05:18.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:18 smithi174 ceph-mon[21825]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-22T22:05:19.274 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:19 smithi192 ceph-mon[22835]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-22T22:05:19.275 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:19 smithi192 ceph-mon[22835]: Cluster is now healthy 2024-04-22T22:05:19.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:19 smithi174 ceph-mon[21825]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-22T22:05:19.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:19 smithi174 ceph-mon[21825]: Cluster is now healthy 2024-04-22T22:05:19.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:19 smithi174 ceph-mon[26730]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-22T22:05:19.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:19 smithi174 ceph-mon[26730]: Cluster is now healthy 2024-04-22T22:05:20.450 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:20 smithi192 ceph-mon[22835]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:05:20.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:20 smithi174 ceph-mon[21825]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:05:20.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:20 smithi174 ceph-mon[26730]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:05:22.444 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:22 smithi192 ceph-mon[22835]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-04-22T22:05:22.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:22 smithi174 ceph-mon[21825]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-04-22T22:05:22.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:22 smithi174 ceph-mon[26730]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-04-22T22:05:23.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:23 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T22:05:23.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:23 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:23.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:23 smithi192 ceph-mon[22835]: Deploying daemon osd.6 on smithi192 2024-04-22T22:05:23.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T22:05:23.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:23 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:23.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:23 smithi174 ceph-mon[21825]: Deploying daemon osd.6 on smithi192 2024-04-22T22:05:23.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:23 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-22T22:05:23.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:23 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:23.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:23 smithi174 ceph-mon[26730]: Deploying daemon osd.6 on smithi192 2024-04-22T22:05:24.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:24 smithi174 ceph-mon[21825]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-22T22:05:24.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:24 smithi174 ceph-mon[26730]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-22T22:05:24.381 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:24 smithi192 ceph-mon[22835]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-04-22T22:05:26.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:26 smithi192 ceph-mon[22835]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-22T22:05:26.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:26 smithi174 ceph-mon[26730]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-22T22:05:26.409 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:26 smithi174 ceph-mon[21825]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-22T22:05:26.815 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:05:26 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:05:26] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:05:27.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:27.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:27.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:27.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:27.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:27.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:27.653 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:27 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:27.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:27.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:27.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:27.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:27.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:27.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:27.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:27.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:27.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:27.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:27.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:27.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:27.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:27.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:27 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:28.533 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:28 smithi192 ceph-mon[22835]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T22:05:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:28 smithi174 ceph-mon[21825]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T22:05:28.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:28 smithi174 ceph-mon[26730]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-04-22T22:05:29.419 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 6 on host 'smithi192' 2024-04-22T22:05:30.082 DEBUG:teuthology.orchestra.run.smithi192:osd.6> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.6.service 2024-04-22T22:05:30.085 INFO:tasks.cephadm:Deploying osd.7 on smithi192 with /dev/vg_nvme/lv_1... 2024-04-22T22:05:30.085 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-22T22:05:30.418 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:05:30.156+0000 7fab8ce0c3c0 -1 osd.6 0 log_to_monitors true 2024-04-22T22:05:30.687 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-mon[22835]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-22T22:05:30.687 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:30.687 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:30.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:30.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:30.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:30.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:30.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:30.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-mon[22835]: Adjusting osd_memory_target on smithi192 to 5354M 2024-04-22T22:05:30.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:30.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:30.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-mon[22835]: from='osd.6 [v2:172.21.15.192:6816/1564782047,v1:172.21.15.192:6817/1564782047]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T22:05:30.688 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:30 smithi192 ceph-mon[22835]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T22:05:30.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[21825]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-22T22:05:30.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:30.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:30.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:30.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:30.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:30.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:30.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:30.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[21825]: Adjusting osd_memory_target on smithi192 to 5354M 2024-04-22T22:05:30.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:30.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:30.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[21825]: from='osd.6 [v2:172.21.15.192:6816/1564782047,v1:172.21.15.192:6817/1564782047]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T22:05:30.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[21825]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T22:05:30.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[26730]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-22T22:05:30.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:30.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:30.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:30.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:30.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:30.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:30.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:30.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[26730]: Adjusting osd_memory_target on smithi192 to 5354M 2024-04-22T22:05:30.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:30.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:30.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[26730]: from='osd.6 [v2:172.21.15.192:6816/1564782047,v1:172.21.15.192:6817/1564782047]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T22:05:30.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:30 smithi174 ceph-mon[26730]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-22T22:05:31.610 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:05:31 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:05:31.478+0000 7fab8380f700 -1 osd.6 0 waiting for initial osdmap 2024-04-22T22:05:31.611 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:05:31 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:05:31.488+0000 7fab7d9a5700 -1 osd.6 43 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T22:05:31.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:31 smithi192 ceph-mon[22835]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T22:05:31.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:31 smithi192 ceph-mon[22835]: osdmap e42: 7 total, 6 up, 7 in 2024-04-22T22:05:31.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:31 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:31.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:31 smithi192 ceph-mon[22835]: from='osd.6 [v2:172.21.15.192:6816/1564782047,v1:172.21.15.192:6817/1564782047]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:31.611 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:31 smithi192 ceph-mon[22835]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:31.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:31 smithi174 ceph-mon[21825]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T22:05:31.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:31 smithi174 ceph-mon[21825]: osdmap e42: 7 total, 6 up, 7 in 2024-04-22T22:05:31.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:31 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:31.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:31 smithi174 ceph-mon[21825]: from='osd.6 [v2:172.21.15.192:6816/1564782047,v1:172.21.15.192:6817/1564782047]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:31.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:31 smithi174 ceph-mon[21825]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:31.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:31 smithi174 ceph-mon[26730]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-22T22:05:31.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:31 smithi174 ceph-mon[26730]: osdmap e42: 7 total, 6 up, 7 in 2024-04-22T22:05:31.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:31 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:31.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:31 smithi174 ceph-mon[26730]: from='osd.6 [v2:172.21.15.192:6816/1564782047,v1:172.21.15.192:6817/1564782047]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:31.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:31 smithi174 ceph-mon[26730]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:31.914 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-22T22:05:31.915 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-22T22:05:31.915 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-04-22T22:05:31.915 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-04-22T22:05:31.915 INFO:teuthology.orchestra.run.smithi192.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0140348 s, 747 MB/s 2024-04-22T22:05:31.915 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 2024-04-22T22:05:31.915 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-04-22T22:05:32.455 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_1 2024-04-22T22:05:32.477 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:32 smithi192 ceph-mon[22835]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:05:32.477 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:32 smithi192 ceph-mon[22835]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-22T22:05:32.477 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:32 smithi192 ceph-mon[22835]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T22:05:32.477 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:32 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:32.477 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:32 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:32.477 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:32 smithi192 ceph-mon[22835]: osd.6 [v2:172.21.15.192:6816/1564782047,v1:172.21.15.192:6817/1564782047] boot 2024-04-22T22:05:32.477 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:32 smithi192 ceph-mon[22835]: osdmap e44: 7 total, 7 up, 7 in 2024-04-22T22:05:32.478 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:32 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:32.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[21825]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:05:32.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[21825]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-22T22:05:32.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[21825]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T22:05:32.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:32.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:32.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[21825]: osd.6 [v2:172.21.15.192:6816/1564782047,v1:172.21.15.192:6817/1564782047] boot 2024-04-22T22:05:32.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[21825]: osdmap e44: 7 total, 7 up, 7 in 2024-04-22T22:05:32.816 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:32.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[26730]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-22T22:05:32.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[26730]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-22T22:05:32.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[26730]: osdmap e43: 7 total, 6 up, 7 in 2024-04-22T22:05:32.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:32.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:32.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[26730]: osd.6 [v2:172.21.15.192:6816/1564782047,v1:172.21.15.192:6817/1564782047] boot 2024-04-22T22:05:32.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[26730]: osdmap e44: 7 total, 7 up, 7 in 2024-04-22T22:05:32.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:32 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:05:34.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:34 smithi192 ceph-mon[22835]: purged_snaps scrub starts 2024-04-22T22:05:34.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:34 smithi192 ceph-mon[22835]: purged_snaps scrub ok 2024-04-22T22:05:34.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:34 smithi192 ceph-mon[22835]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:34.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:34 smithi192 ceph-mon[22835]: osdmap e45: 7 total, 7 up, 7 in 2024-04-22T22:05:34.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:34 smithi174 ceph-mon[21825]: purged_snaps scrub starts 2024-04-22T22:05:34.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:34 smithi174 ceph-mon[21825]: purged_snaps scrub ok 2024-04-22T22:05:34.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:34 smithi174 ceph-mon[21825]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:34.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:34 smithi174 ceph-mon[21825]: osdmap e45: 7 total, 7 up, 7 in 2024-04-22T22:05:34.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:34 smithi174 ceph-mon[26730]: purged_snaps scrub starts 2024-04-22T22:05:34.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:34 smithi174 ceph-mon[26730]: purged_snaps scrub ok 2024-04-22T22:05:34.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:34 smithi174 ceph-mon[26730]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:34.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:34 smithi174 ceph-mon[26730]: osdmap e45: 7 total, 7 up, 7 in 2024-04-22T22:05:35.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:35 smithi174 ceph-mon[26730]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T22:05:35.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:35 smithi174 ceph-mon[26730]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:05:35.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:35 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:05:35.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:35 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:05:35.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:35 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:35.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:35 smithi174 ceph-mon[21825]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T22:05:35.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:35 smithi174 ceph-mon[21825]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:05:35.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:35 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:05:35.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:35 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:05:35.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:35 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:35.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:35 smithi192 ceph-mon[22835]: osdmap e46: 7 total, 7 up, 7 in 2024-04-22T22:05:35.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:35 smithi192 ceph-mon[22835]: from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:05:35.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:35 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-22T22:05:35.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:35 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-22T22:05:35.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:35 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:36.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:36 smithi192 ceph-mon[22835]: pgmap v150: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T22:05:36.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:36 smithi192 ceph-mon[22835]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T22:05:36.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:36 smithi174 ceph-mon[21825]: pgmap v150: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T22:05:36.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:36 smithi174 ceph-mon[21825]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T22:05:36.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:36 smithi174 ceph-mon[26730]: pgmap v150: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-22T22:05:36.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:36 smithi174 ceph-mon[26730]: osdmap e47: 7 total, 7 up, 7 in 2024-04-22T22:05:37.033 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:05:36 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:05:36] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:05:37.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[21825]: osdmap e48: 7 total, 7 up, 7 in 2024-04-22T22:05:37.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[21825]: from='client.? 172.21.15.192:0/3104051520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a033a707-0d56-4b05-8779-a85d5f337880"}]: dispatch 2024-04-22T22:05:37.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[21825]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a033a707-0d56-4b05-8779-a85d5f337880"}]: dispatch 2024-04-22T22:05:37.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[21825]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a033a707-0d56-4b05-8779-a85d5f337880"}]': finished 2024-04-22T22:05:37.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[21825]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T22:05:37.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:37.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[26730]: osdmap e48: 7 total, 7 up, 7 in 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[26730]: from='client.? 172.21.15.192:0/3104051520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a033a707-0d56-4b05-8779-a85d5f337880"}]: dispatch 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[26730]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a033a707-0d56-4b05-8779-a85d5f337880"}]: dispatch 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[26730]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a033a707-0d56-4b05-8779-a85d5f337880"}]': finished 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[26730]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:37.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:37.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:37.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:37.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:37.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:37 smithi192 ceph-mon[22835]: osdmap e48: 7 total, 7 up, 7 in 2024-04-22T22:05:37.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:37 smithi192 ceph-mon[22835]: from='client.? 172.21.15.192:0/3104051520' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a033a707-0d56-4b05-8779-a85d5f337880"}]: dispatch 2024-04-22T22:05:37.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:37 smithi192 ceph-mon[22835]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a033a707-0d56-4b05-8779-a85d5f337880"}]: dispatch 2024-04-22T22:05:37.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:37 smithi192 ceph-mon[22835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a033a707-0d56-4b05-8779-a85d5f337880"}]': finished 2024-04-22T22:05:37.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:37 smithi192 ceph-mon[22835]: osdmap e49: 8 total, 7 up, 8 in 2024-04-22T22:05:37.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:37 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:37.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:37 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:37.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:37 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:37.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:37 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:37.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:37 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:37.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:37 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:37.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:37 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:38.054 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:38 smithi192 ceph-mon[22835]: pgmap v154: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T22:05:38.055 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:38 smithi192 ceph-mon[22835]: Detected new or changed devices on smithi192 2024-04-22T22:05:38.055 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:38 smithi192 ceph-mon[22835]: pgmap v155: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 191 KiB/s, 0 objects/s recovering 2024-04-22T22:05:38.055 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:38 smithi192 ceph-mon[22835]: from='client.? 172.21.15.192:0/649374663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:05:38.055 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:38 smithi192 ceph-mon[22835]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-22T22:05:38.055 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:38 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:38.055 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:38 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:38.055 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:38 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:38.055 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:38 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:38.055 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:38 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:38.055 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:38 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:38.065 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:05:37 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:05:37] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:05:38.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[21825]: pgmap v154: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T22:05:38.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[21825]: Detected new or changed devices on smithi192 2024-04-22T22:05:38.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[21825]: pgmap v155: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 191 KiB/s, 0 objects/s recovering 2024-04-22T22:05:38.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[21825]: from='client.? 172.21.15.192:0/649374663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:05:38.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[21825]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-22T22:05:38.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:38.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:38.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:38.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:38.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:38.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:38.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[26730]: pgmap v154: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-22T22:05:38.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[26730]: Detected new or changed devices on smithi192 2024-04-22T22:05:38.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[26730]: pgmap v155: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail; 191 KiB/s, 0 objects/s recovering 2024-04-22T22:05:38.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[26730]: from='client.? 172.21.15.192:0/649374663' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-22T22:05:38.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[26730]: Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-22T22:05:38.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:38.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:38.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:38.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:38.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:38.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:38 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:39.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:39 smithi192 ceph-mon[22835]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:39.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:39 smithi192 ceph-mon[22835]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-22T22:05:39.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:39 smithi192 ceph-mon[22835]: Cluster is now healthy 2024-04-22T22:05:39.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:39 smithi174 ceph-mon[21825]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:39.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:39 smithi174 ceph-mon[21825]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-22T22:05:39.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:39 smithi174 ceph-mon[21825]: Cluster is now healthy 2024-04-22T22:05:39.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:39 smithi174 ceph-mon[26730]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:39.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:39 smithi174 ceph-mon[26730]: Health check cleared: CEPHADM_STRAY_DAEMON (was: 1 stray daemon(s) not managed by cephadm) 2024-04-22T22:05:39.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:39 smithi174 ceph-mon[26730]: Cluster is now healthy 2024-04-22T22:05:41.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:41 smithi174 ceph-mon[21825]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:41.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:41 smithi174 ceph-mon[26730]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:41.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:41 smithi192 ceph-mon[22835]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:43.204 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:43 smithi192 ceph-mon[22835]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:43.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:43 smithi174 ceph-mon[26730]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:43.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:43 smithi174 ceph-mon[21825]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:44.208 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T22:05:44.208 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:44 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:44.208 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:44 smithi192 ceph-mon[22835]: Deploying daemon osd.7 on smithi192 2024-04-22T22:05:44.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T22:05:44.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:44 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:44.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:44 smithi174 ceph-mon[21825]: Deploying daemon osd.7 on smithi192 2024-04-22T22:05:44.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-22T22:05:44.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:44 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:44.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:44 smithi174 ceph-mon[26730]: Deploying daemon osd.7 on smithi192 2024-04-22T22:05:45.275 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:45 smithi192 ceph-mon[22835]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:45.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:45 smithi174 ceph-mon[21825]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:45.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:45 smithi174 ceph-mon[26730]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:46.667 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:05:46 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:05:46] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:05:47.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:47 smithi174 ceph-mon[21825]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:47.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:47 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:47.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:47 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:47.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:47 smithi174 ceph-mon[26730]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:47.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:47 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:47.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:47 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:47.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:47 smithi192 ceph-mon[22835]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:47.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:47 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:47.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:47 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:49.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:49 smithi192 ceph-mon[22835]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:49.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:49 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:49.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:49 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:49.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:49 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:49.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:49 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:49.628 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:49 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:49.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:49 smithi174 ceph-mon[21825]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:49.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:49 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:49.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:49 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:49.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:49 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:49.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:49 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:49.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:49 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:49.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:49 smithi174 ceph-mon[26730]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:49.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:49 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:49.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:49 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:49.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:49 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:49.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:49 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:49.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:49 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:51.052 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 7 on host 'smithi192' 2024-04-22T22:05:51.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:51 smithi192 ceph-mon[22835]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:51.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:51 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:51.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:51 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:51.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:51 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:51.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:51 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:51.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:51 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:51.664 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:51 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:51.665 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:51 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:51.665 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:51 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:51.665 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:51 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:51.665 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:51 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:51.686 DEBUG:teuthology.orchestra.run.smithi192:osd.7> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.7.service 2024-04-22T22:05:51.729 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-22T22:05:51.729 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T22:05:51.759 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[21825]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:51.759 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:51.759 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:51.759 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:51.759 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:51.759 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:51.759 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:51.759 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:51.759 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:51.759 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:51.759 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:51.760 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[26730]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:51.760 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:51.760 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:51.760 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:51.760 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:51.760 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:51.760 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:51.760 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:51.760 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:05:51.760 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:51.760 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:51 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:52.753 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:52 smithi174 ceph-mon[21825]: Adjusting osd_memory_target on smithi192 to 4016M 2024-04-22T22:05:52.754 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:52 smithi174 ceph-mon[21825]: from='osd.7 [v2:172.21.15.192:6824/1603978747,v1:172.21.15.192:6825/1603978747]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T22:05:52.754 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:52 smithi174 ceph-mon[21825]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T22:05:52.754 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:52 smithi174 ceph-mon[26730]: Adjusting osd_memory_target on smithi192 to 4016M 2024-04-22T22:05:52.754 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:52 smithi174 ceph-mon[26730]: from='osd.7 [v2:172.21.15.192:6824/1603978747,v1:172.21.15.192:6825/1603978747]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T22:05:52.755 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:52 smithi174 ceph-mon[26730]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T22:05:52.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:52 smithi192 ceph-mon[22835]: Adjusting osd_memory_target on smithi192 to 4016M 2024-04-22T22:05:52.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:52 smithi192 ceph-mon[22835]: from='osd.7 [v2:172.21.15.192:6824/1603978747,v1:172.21.15.192:6825/1603978747]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T22:05:52.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:52 smithi192 ceph-mon[22835]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-22T22:05:53.435 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:53 smithi174 ceph-mon[21825]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:53.436 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:53 smithi174 ceph-mon[21825]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T22:05:53.436 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:53 smithi174 ceph-mon[21825]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T22:05:53.436 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:53 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:53.436 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:53 smithi174 ceph-mon[21825]: from='osd.7 [v2:172.21.15.192:6824/1603978747,v1:172.21.15.192:6825/1603978747]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:53.436 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:53 smithi174 ceph-mon[21825]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:53.436 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:53 smithi174 ceph-mon[26730]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:53.436 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:53 smithi174 ceph-mon[26730]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T22:05:53.436 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:53 smithi174 ceph-mon[26730]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T22:05:53.437 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:53 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:53.437 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:53 smithi174 ceph-mon[26730]: from='osd.7 [v2:172.21.15.192:6824/1603978747,v1:172.21.15.192:6825/1603978747]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:53.437 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:53 smithi174 ceph-mon[26730]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:53.792 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T22:05:53.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:53 smithi192 ceph-mon[22835]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:53.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:53 smithi192 ceph-mon[22835]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-22T22:05:53.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:53 smithi192 ceph-mon[22835]: osdmap e50: 8 total, 7 up, 8 in 2024-04-22T22:05:53.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:53 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:53.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:53 smithi192 ceph-mon[22835]: from='osd.7 [v2:172.21.15.192:6824/1603978747,v1:172.21.15.192:6825/1603978747]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:53.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:53 smithi192 ceph-mon[22835]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-04-22T22:05:53.816 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:05:53 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:05:53.370+0000 7f7f29cf7700 -1 osd.7 0 waiting for initial osdmap 2024-04-22T22:05:53.816 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:05:53 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:05:53.381+0000 7f7f26692700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-22T22:05:54.378 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[21825]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-22T22:05:54.378 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[21825]: osdmap e51: 8 total, 7 up, 8 in 2024-04-22T22:05:54.378 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:54.378 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:54.378 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:05:54.378 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:05:54.378 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1730252854' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T22:05:54.379 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[26730]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-22T22:05:54.379 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[26730]: osdmap e51: 8 total, 7 up, 8 in 2024-04-22T22:05:54.379 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:54.379 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:54.379 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:05:54.379 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:05:54.379 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:54 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1730252854' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T22:05:54.400 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":51,"num_osds":8,"num_up_osds":7,"osd_up_since":1713823532,"num_in_osds":8,"osd_in_since":1713823536,"num_remapped_pgs":0} 2024-04-22T22:05:54.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:54 smithi192 ceph-mon[22835]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-04-22T22:05:54.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:54 smithi192 ceph-mon[22835]: osdmap e51: 8 total, 7 up, 8 in 2024-04-22T22:05:54.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:54.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:54.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:05:54.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:54 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:05:54.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:54 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1730252854' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T22:05:55.402 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-22T22:05:55.713 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[21825]: purged_snaps scrub starts 2024-04-22T22:05:55.713 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[21825]: purged_snaps scrub ok 2024-04-22T22:05:55.713 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[21825]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:55.713 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:55.713 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[21825]: osd.7 [v2:172.21.15.192:6824/1603978747,v1:172.21.15.192:6825/1603978747] boot 2024-04-22T22:05:55.713 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[21825]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T22:05:55.713 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:55.714 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[26730]: purged_snaps scrub starts 2024-04-22T22:05:55.714 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[26730]: purged_snaps scrub ok 2024-04-22T22:05:55.714 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[26730]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:55.714 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:55.714 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[26730]: osd.7 [v2:172.21.15.192:6824/1603978747,v1:172.21.15.192:6825/1603978747] boot 2024-04-22T22:05:55.714 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[26730]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T22:05:55.714 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:55 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:55.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:55 smithi192 ceph-mon[22835]: purged_snaps scrub starts 2024-04-22T22:05:55.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:55 smithi192 ceph-mon[22835]: purged_snaps scrub ok 2024-04-22T22:05:55.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:55 smithi192 ceph-mon[22835]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-22T22:05:55.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:55.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:55 smithi192 ceph-mon[22835]: osd.7 [v2:172.21.15.192:6824/1603978747,v1:172.21.15.192:6825/1603978747] boot 2024-04-22T22:05:55.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:55 smithi192 ceph-mon[22835]: osdmap e52: 8 total, 8 up, 8 in 2024-04-22T22:05:55.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:55 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:05:56.617 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:56 smithi174 ceph-mon[21825]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T22:05:56.617 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:56 smithi174 ceph-mon[26730]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T22:05:56.788 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:56 smithi192 ceph-mon[22835]: osdmap e53: 8 total, 8 up, 8 in 2024-04-22T22:05:57.262 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T22:05:57.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:57 smithi174 ceph-mon[26730]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:05:57.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:57 smithi174 ceph-mon[26730]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T22:05:57.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:57 smithi174 ceph-mon[26730]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T22:05:57.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:57 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2613873048' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T22:05:57.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:57 smithi174 ceph-mon[21825]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:05:57.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:57 smithi174 ceph-mon[21825]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T22:05:57.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:57 smithi174 ceph-mon[21825]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T22:05:57.566 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:57 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2613873048' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T22:05:57.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:57 smithi192 ceph-mon[22835]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:05:57.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:57 smithi192 ceph-mon[22835]: osdmap e54: 8 total, 8 up, 8 in 2024-04-22T22:05:57.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:57 smithi192 ceph-mon[22835]: osdmap e55: 8 total, 8 up, 8 in 2024-04-22T22:05:57.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:57 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2613873048' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-22T22:05:57.837 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1713823554,"num_in_osds":8,"osd_in_since":1713823536,"num_remapped_pgs":0} 2024-04-22T22:05:57.838 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T22:05:58.052 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:05:58.065 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:05:57 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:05:57] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:05:58.401 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:05:58] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:05:59.136 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:05:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: Detected new or changed devices on smithi192 2024-04-22T22:05:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:05:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: osdmap e56: 8 total, 8 up, 8 in 2024-04-22T22:05:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:59.137 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:59.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:59.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:05:59.138 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:58 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.222 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:05:59.222 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: Detected new or changed devices on smithi192 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: osdmap e56: 8 total, 8 up, 8 in 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:59.223 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:05:59.224 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.224 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:05:59.224 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.224 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: Detected new or changed devices on smithi192 2024-04-22T22:05:59.224 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.224 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:59.224 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:59.224 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:59.224 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 46 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:05:59.224 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.224 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: osdmap e56: 8 total, 8 up, 8 in 2024-04-22T22:05:59.224 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.225 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.225 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:05:59.225 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:05:59.225 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:05:59.225 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:05:59.225 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:58 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:05:59.861 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T22:05:59.862 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":56,"fsid":"b7cbfdbc-00f3-11ef-bc93-c7b262605968","created":"2024-04-22T22:01:07.496253+0000","modified":"2024-04-22T22:05:58.043737+0000","last_up_change":"2024-04-22T22:05:54.368445+0000","last_in_change":"2024-04-22T22:05:36.482223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T22:04:05.963559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"10b65a80-27ba-4f98-b4a5-46610cbcee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2571929691},{"type":"v1","addr":"172.21.15.174:6803","nonce":2571929691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2571929691},{"type":"v1","addr":"172.21.15.174:6805","nonce":2571929691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2571929691},{"type":"v1","addr":"172.21.15.174:6809","nonce":2571929691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2571929691},{"type":"v1","addr":"172.21.15.174:6807","nonce":2571929691}]},"public_addr":"172.21.15.174:6803/2571929691","cluster_addr":"172.21.15.174:6805/2571929691","heartbeat_back_addr":"172.21.15.174:6809/2571929691","heartbeat_front_addr":"172.21.15.174:6807/2571929691","state":["exists","up"]},{"osd":1,"uuid":"b4d7e7fc-0a42-4a61-ba4f-a1693dc2a970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":222655237},{"type":"v1","addr":"172.21.15.174:6811","nonce":222655237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":222655237},{"type":"v1","addr":"172.21.15.174:6813","nonce":222655237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":222655237},{"type":"v1","addr":"172.21.15.174:6817","nonce":222655237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":222655237},{"type":"v1","addr":"172.21.15.174:6815","nonce":222655237}]},"public_addr":"172.21.15.174:6811/222655237","cluster_addr":"172.21.15.174:6813/222655237","heartbeat_back_addr":"172.21.15.174:6817/222655237","heartbeat_front_addr":"172.21.15.174:6815/222655237","state":["exists","up"]},{"osd":2,"uuid":"439d2d85-e8b4-4a98-bc6d-11fcab19c75d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":77924542},{"type":"v1","addr":"172.21.15.174:6819","nonce":77924542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":77924542},{"type":"v1","addr":"172.21.15.174:6821","nonce":77924542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":77924542},{"type":"v1","addr":"172.21.15.174:6825","nonce":77924542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":77924542},{"type":"v1","addr":"172.21.15.174:6823","nonce":77924542}]},"public_addr":"172.21.15.174:6819/77924542","cluster_addr":"172.21.15.174:6821/77924542","heartbeat_back_addr":"172.21.15.174:6825/77924542","heartbeat_front_addr":"172.21.15.174:6823/77924542","state":["exists","up"]},{"osd":3,"uuid":"d0ce25dc-969a-4435-ab08-96d78791c719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1588342953},{"type":"v1","addr":"172.21.15.174:6827","nonce":1588342953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1588342953},{"type":"v1","addr":"172.21.15.174:6829","nonce":1588342953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1588342953},{"type":"v1","addr":"172.21.15.174:6833","nonce":1588342953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1588342953},{"type":"v1","addr":"172.21.15.174:6831","nonce":1588342953}]},"public_addr":"172.21.15.174:6827/1588342953","cluster_addr":"172.21.15.174:6829/1588342953","heartbeat_back_addr":"172.21.15.174:6833/1588342953","heartbeat_front_addr":"172.21.15.174:6831/1588342953","state":["exists","up"]},{"osd":4,"uuid":"b95a43b7-c458-4f35-8494-52af4cc55af8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":469106314},{"type":"v1","addr":"172.21.15.192:6801","nonce":469106314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":469106314},{"type":"v1","addr":"172.21.15.192:6803","nonce":469106314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":469106314},{"type":"v1","addr":"172.21.15.192:6807","nonce":469106314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":469106314},{"type":"v1","addr":"172.21.15.192:6805","nonce":469106314}]},"public_addr":"172.21.15.192:6801/469106314","cluster_addr":"172.21.15.192:6803/469106314","heartbeat_back_addr":"172.21.15.192:6807/469106314","heartbeat_front_addr":"172.21.15.192:6805/469106314","state":["exists","up"]},{"osd":5,"uuid":"3400fe78-1500-44a8-a526-2a80969d5c54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":645383067},{"type":"v1","addr":"172.21.15.192:6809","nonce":645383067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":645383067},{"type":"v1","addr":"172.21.15.192:6811","nonce":645383067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":645383067},{"type":"v1","addr":"172.21.15.192:6815","nonce":645383067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":645383067},{"type":"v1","addr":"172.21.15.192:6813","nonce":645383067}]},"public_addr":"172.21.15.192:6809/645383067","cluster_addr":"172.21.15.192:6811/645383067","heartbeat_back_addr":"172.21.15.192:6815/645383067","heartbeat_front_addr":"172.21.15.192:6813/645383067","state":["exists","up"]},{"osd":6,"uuid":"f446ce85-1bc3-4f2c-a013-7841180ee701","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1564782047},{"type":"v1","addr":"172.21.15.192:6817","nonce":1564782047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1564782047},{"type":"v1","addr":"172.21.15.192:6819","nonce":1564782047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1564782047},{"type":"v1","addr":"172.21.15.192:6823","nonce":1564782047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1564782047},{"type":"v1","addr":"172.21.15.192:6821","nonce":1564782047}]},"public_addr":"172.21.15.192:6817/1564782047","cluster_addr":"172.21.15.192:6819/1564782047","heartbeat_back_addr":"172.21.15.192:6823/1564782047","heartbeat_front_addr":"172.21.15.192:6821/1564782047","state":["exists","up"]},{"osd":7,"uuid":"a033a707-0d56-4b05-8779-a85d5f337880","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":1603978747},{"type":"v1","addr":"172.21.15.192:6825","nonce":1603978747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":1603978747},{"type":"v1","addr":"172.21.15.192:6827","nonce":1603978747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":1603978747},{"type":"v1","addr":"172.21.15.192:6831","nonce":1603978747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":1603978747},{"type":"v1","addr":"172.21.15.192:6829","nonce":1603978747}]},"public_addr":"172.21.15.192:6825/1603978747","cluster_addr":"172.21.15.192:6827/1603978747","heartbeat_back_addr":"172.21.15.192:6831/1603978747","heartbeat_front_addr":"172.21.15.192:6829/1603978747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:03:16.803033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:03:40.119346+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:04:03.225265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:04:27.029259+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:04:49.376150+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:05:10.815050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:05:31.203539+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:05:52.478808+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.174:0/3592746357":"2024-04-23T22:01:53.599383+0000","172.21.15.174:6801/2262747749":"2024-04-23T22:01:53.599383+0000","172.21.15.174:0/4157104731":"2024-04-23T22:01:53.599383+0000","172.21.15.174:6800/2262747749":"2024-04-23T22:01:53.599383+0000","172.21.15.174:0/3967906832":"2024-04-23T22:01:53.599383+0000","172.21.15.174:0/1922808348":"2024-04-23T22:01:31.026657+0000","172.21.15.174:0/2470543672":"2024-04-23T22:01:31.026657+0000","172.21.15.174:0/1468603511":"2024-04-23T22:01:31.026657+0000","172.21.15.174:6800/4261879208":"2024-04-23T22:01:31.026657+0000","172.21.15.174:6801/4261879208":"2024-04-23T22:01:31.026657+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T22:06:00.002 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:05:59 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2652568142' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:06:00.186 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:05:59 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2652568142' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:06:00.187 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:05:59 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2652568142' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:06:00.485 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-22T22:04:05.963559+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-22T22:06:00.485 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-22T22:06:00.692 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:06:00.947 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:00 smithi192 ceph-mon[22835]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:01.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:00 smithi174 ceph-mon[21825]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:01.065 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:00 smithi174 ceph-mon[26730]: pgmap v176: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:02.505 INFO:teuthology.orchestra.run.smithi174.stdout:pg_num: 1 2024-04-22T22:06:03.149 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:03 smithi174 ceph-mon[21825]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-22T22:06:03.149 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:03 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/4266367204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T22:06:03.149 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:03 smithi174 ceph-mon[26730]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-22T22:06:03.149 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:03 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/4266367204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T22:06:03.173 INFO:tasks.cephadm:Adding prometheus.a on smithi192 2024-04-22T22:06:03.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch apply prometheus '1;smithi192=a' 2024-04-22T22:06:03.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:03 smithi192 ceph-mon[22835]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-04-22T22:06:03.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:03 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/4266367204' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-22T22:06:05.039 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled prometheus update... 2024-04-22T22:06:05.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:05 smithi192 ceph-mon[22835]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-22T22:06:05.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:05 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:06:05.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:05 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:05.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:05 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:05.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:05 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:05.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:05 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:06:05.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:05 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T22:06:05.437 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[26730]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-22T22:06:05.437 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:06:05.437 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:05.437 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:05.437 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:05.437 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:06:05.437 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T22:06:05.438 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[21825]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-04-22T22:06:05.438 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:06:05.438 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:05.438 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:05.438 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:05.438 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' 2024-04-22T22:06:05.438 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:05 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T22:06:05.675 DEBUG:teuthology.orchestra.run.smithi192:prometheus.a> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@prometheus.a.service 2024-04-22T22:06:05.678 INFO:tasks.cephadm:Adding node-exporter.a on smithi174 2024-04-22T22:06:05.678 INFO:tasks.cephadm:Adding node-exporter.b on smithi192 2024-04-22T22:06:05.678 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch apply node-exporter '2;smithi174=a;smithi192=b' 2024-04-22T22:06:06.163 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ignoring --setuser ceph since I am not root 2024-04-22T22:06:06.163 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ignoring --setgroup ceph since I am not root 2024-04-22T22:06:06.163 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-mgr[22063]: -- 172.21.15.174:0/666555314 <== mon.1 v2:172.21.15.174:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f791292340 con 0x55f79126b000 2024-04-22T22:06:06.262 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:06 smithi192 ceph-mon[22835]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:06:06.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:06 smithi192 ceph-mon[22835]: Saving service prometheus spec with placement smithi192=a;count:1 2024-04-22T22:06:06.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:06 smithi192 ceph-mon[22835]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T22:06:06.263 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:06 smithi192 ceph-mon[22835]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-22T22:06:06.263 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:06 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ignoring --setuser ceph since I am not root 2024-04-22T22:06:06.263 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:06 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ignoring --setgroup ceph since I am not root 2024-04-22T22:06:06.263 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:06 smithi192 ceph-mgr[24377]: -- 172.21.15.192:0/3290357459 <== mon.2 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d194dce340 con 0x55d194da6800 2024-04-22T22:06:06.263 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:06 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:06.286+0000 7fac7f216000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T22:06:06.434 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-mon[21825]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:06:06.434 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-mon[21825]: Saving service prometheus spec with placement smithi192=a;count:1 2024-04-22T22:06:06.434 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-mon[21825]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T22:06:06.434 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-mon[21825]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-22T22:06:06.435 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:06.293+0000 7f9f60b40000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-22T22:06:06.435 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:06.427+0000 7f9f60b40000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T22:06:06.435 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-mon[26730]: from='client.34313 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:06:06.435 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-mon[26730]: Saving service prometheus spec with placement smithi192=a;count:1 2024-04-22T22:06:06.435 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-mon[26730]: from='mgr.14150 172.21.15.174:0/3991344232' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T22:06:06.435 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-mon[26730]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-22T22:06:06.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:06 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:06.414+0000 7fac7f216000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-22T22:06:06.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:06 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:06.512+0000 7fac7f216000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T22:06:06.753 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:06.527+0000 7f9f60b40000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-22T22:06:07.010 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:06 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:06.824+0000 7fac7f216000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T22:06:07.010 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:06 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:06.908+0000 7fac7f216000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T22:06:07.065 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:06.845+0000 7f9f60b40000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-22T22:06:07.065 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:06 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:06.929+0000 7f9f60b40000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-22T22:06:08.217 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:08 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:08.077+0000 7f9f60b40000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T22:06:08.315 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:08 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:08.041+0000 7fac7f216000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-22T22:06:08.315 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:08 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:08.202+0000 7fac7f216000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T22:06:08.315 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:08 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:08.283+0000 7fac7f216000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T22:06:08.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:08 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:08.241+0000 7f9f60b40000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-22T22:06:08.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:08 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:08.323+0000 7f9f60b40000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-22T22:06:08.982 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:08 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:08.591+0000 7fac7f216000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T22:06:08.983 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:08 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:08.679+0000 7fac7f216000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T22:06:08.983 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:08 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:08.768+0000 7fac7f216000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T22:06:08.983 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:09 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:09.006+0000 7fac7f216000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T22:06:09.038 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:08 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:08.634+0000 7f9f60b40000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-22T22:06:09.038 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:08 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:08.723+0000 7f9f60b40000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-22T22:06:09.038 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:08 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:08.816+0000 7f9f60b40000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-22T22:06:09.038 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:09 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:09.062+0000 7f9f60b40000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-22T22:06:09.815 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:09 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:09.436+0000 7fac7f216000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T22:06:09.815 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:09 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:09.494+0000 7f9f60b40000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-22T22:06:10.293 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:09 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:09.938+0000 7fac7f216000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T22:06:10.293 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:10 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:10.030+0000 7fac7f216000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T22:06:10.314 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:09 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:09.974+0000 7f9f60b40000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-22T22:06:10.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:10 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:10.066+0000 7f9f60b40000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-22T22:06:10.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:10 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:10.317+0000 7fac7f216000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T22:06:10.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:10 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:10.405+0000 7fac7f216000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T22:06:10.815 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:10 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:10.357+0000 7f9f60b40000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-22T22:06:10.815 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:10 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:10.445+0000 7f9f60b40000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-22T22:06:11.065 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:10 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:10.804+0000 7fac7f216000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T22:06:11.065 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:10 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:10.934+0000 7fac7f216000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T22:06:11.065 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:11.025+0000 7fac7f216000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T22:06:11.155 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:10 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:10.841+0000 7f9f60b40000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-22T22:06:11.156 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:10 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:10.973+0000 7f9f60b40000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-22T22:06:11.156 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:11 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:11.065+0000 7f9f60b40000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-22T22:06:11.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:11 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:11.179+0000 7f9f60b40000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T22:06:11.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:11 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:11.271+0000 7f9f60b40000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T22:06:11.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:11.138+0000 7fac7f216000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-22T22:06:11.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:11.225+0000 7fac7f216000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-22T22:06:12.065 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:11.761+0000 7fac7f216000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T22:06:12.065 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:11 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:11.818+0000 7f9f60b40000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-22T22:06:12.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:12 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:12.181+0000 7f9f60b40000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T22:06:12.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:12.120+0000 7fac7f216000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-22T22:06:12.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: [22/Apr/2024:22:06:12] ENGINE Bus STARTING 2024-04-22T22:06:12.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: CherryPy Checker: 2024-04-22T22:06:12.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: The Application mounted at '' has an empty config. 2024-04-22T22:06:12.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: 2024-04-22T22:06:12.566 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: [22/Apr/2024:22:06:12] ENGINE Serving on http://:::9283 2024-04-22T22:06:12.566 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:06:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: [22/Apr/2024:22:06:12] ENGINE Bus STARTED 2024-04-22T22:06:13.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[21825]: Standby manager daemon x restarted 2024-04-22T22:06:13.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[21825]: Standby manager daemon x started 2024-04-22T22:06:13.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[21825]: from='mgr.? 172.21.15.192:0/4031389384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-22T22:06:13.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[21825]: from='mgr.? 172.21.15.192:0/4031389384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T22:06:13.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[21825]: from='mgr.? 172.21.15.192:0/4031389384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-22T22:06:13.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[21825]: from='mgr.? 172.21.15.192:0/4031389384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T22:06:13.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[21825]: Active manager daemon y restarted 2024-04-22T22:06:13.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[21825]: Activating manager daemon y 2024-04-22T22:06:13.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[21825]: osdmap e57: 8 total, 8 up, 8 in 2024-04-22T22:06:13.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[26730]: Standby manager daemon x restarted 2024-04-22T22:06:13.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[26730]: Standby manager daemon x started 2024-04-22T22:06:13.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[26730]: from='mgr.? 172.21.15.192:0/4031389384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-22T22:06:13.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[26730]: from='mgr.? 172.21.15.192:0/4031389384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T22:06:13.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[26730]: from='mgr.? 172.21.15.192:0/4031389384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-22T22:06:13.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[26730]: from='mgr.? 172.21.15.192:0/4031389384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T22:06:13.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[26730]: Active manager daemon y restarted 2024-04-22T22:06:13.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[26730]: Activating manager daemon y 2024-04-22T22:06:13.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-mon[26730]: osdmap e57: 8 total, 8 up, 8 in 2024-04-22T22:06:13.317 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: [22/Apr/2024:22:06:13] ENGINE Bus STARTING 2024-04-22T22:06:13.317 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: CherryPy Checker: 2024-04-22T22:06:13.317 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: The Application mounted at '' has an empty config. 2024-04-22T22:06:13.317 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:06:13.317 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: [22/Apr/2024:22:06:13] ENGINE Serving on http://:::9283 2024-04-22T22:06:13.317 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: [22/Apr/2024:22:06:13] ENGINE Bus STARTED 2024-04-22T22:06:13.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:13 smithi192 ceph-mon[22835]: Standby manager daemon x restarted 2024-04-22T22:06:13.317 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:13 smithi192 ceph-mon[22835]: Standby manager daemon x started 2024-04-22T22:06:13.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:13 smithi192 ceph-mon[22835]: from='mgr.? 172.21.15.192:0/4031389384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-22T22:06:13.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:13 smithi192 ceph-mon[22835]: from='mgr.? 172.21.15.192:0/4031389384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-22T22:06:13.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:13 smithi192 ceph-mon[22835]: from='mgr.? 172.21.15.192:0/4031389384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-22T22:06:13.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:13 smithi192 ceph-mon[22835]: from='mgr.? 172.21.15.192:0/4031389384' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-22T22:06:13.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:13 smithi192 ceph-mon[22835]: Active manager daemon y restarted 2024-04-22T22:06:13.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:13 smithi192 ceph-mon[22835]: Activating manager daemon y 2024-04-22T22:06:13.318 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:13 smithi192 ceph-mon[22835]: osdmap e57: 8 total, 8 up, 8 in 2024-04-22T22:06:13.600 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: [22/Apr/2024:22:06:13] ENGINE Bus STARTING 2024-04-22T22:06:14.024 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: [22/Apr/2024:22:06:13] ENGINE Serving on https://172.21.15.174:7150 2024-04-22T22:06:14.024 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:13 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: [22/Apr/2024:22:06:13] ENGINE Bus STARTED 2024-04-22T22:06:14.053 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled node-exporter update... 2024-04-22T22:06:14.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: mgrmap e17: y(active, starting, since 0.866251s), standbys: x 2024-04-22T22:06:14.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:06:14.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:06:14.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:06:14.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T22:06:14.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: Manager daemon y is now available 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:14.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:14.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:14.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:14.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:06:14.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:06:14.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]': finished 2024-04-22T22:06:14.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-22T22:06:14.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: mgrmap e17: y(active, starting, since 0.866251s), standbys: x 2024-04-22T22:06:14.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:06:14.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:06:14.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:06:14.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T22:06:14.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T22:06:14.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:06:14.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:06:14.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:06:14.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:06:14.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:06:14.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:06:14.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:06:14.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:06:14.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T22:06:14.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T22:06:14.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T22:06:14.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: Manager daemon y is now available 2024-04-22T22:06:14.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:14.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:14.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:14.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:14.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:06:14.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:06:14.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]': finished 2024-04-22T22:06:14.319 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-22T22:06:14.320 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]': finished 2024-04-22T22:06:14.320 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: Updating smithi192:/etc/ceph/ceph.conf 2024-04-22T22:06:14.320 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.320 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:06:14.320 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:06:14.320 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.320 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.320 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.320 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T22:06:14.320 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]': finished 2024-04-22T22:06:14.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: Updating smithi192:/etc/ceph/ceph.conf 2024-04-22T22:06:14.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:06:14.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:06:14.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T22:06:14.321 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: mgrmap e17: y(active, starting, since 0.866251s), standbys: x 2024-04-22T22:06:14.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-22T22:06:14.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-22T22:06:14.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-22T22:06:14.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-22T22:06:14.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-22T22:06:14.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-22T22:06:14.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-22T22:06:14.322 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: Manager daemon y is now available 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:14.323 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:06:14.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-04-22T22:06:14.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi174", "name": "osd_memory_target"}]': finished 2024-04-22T22:06:14.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: Updating smithi174:/etc/ceph/ceph.conf 2024-04-22T22:06:14.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]': finished 2024-04-22T22:06:14.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: Updating smithi192:/etc/ceph/ceph.conf 2024-04-22T22:06:14.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:06:14.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:06:14.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-22T22:06:14.324 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:14.557 DEBUG:teuthology.orchestra.run.smithi174:node-exporter.a> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@node-exporter.a.service 2024-04-22T22:06:14.560 DEBUG:teuthology.orchestra.run.smithi192:node-exporter.b> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@node-exporter.b.service 2024-04-22T22:06:14.629 INFO:tasks.cephadm:Adding alertmanager.a on smithi174 2024-04-22T22:06:14.630 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch apply alertmanager '1;smithi174=a' 2024-04-22T22:06:15.279 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:15 smithi192 ceph-mon[22835]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:06:15.280 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:15 smithi192 ceph-mon[22835]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:06:15.280 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:15 smithi192 ceph-mon[22835]: [22/Apr/2024:22:06:13] ENGINE Bus STARTING 2024-04-22T22:06:15.280 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:15 smithi192 ceph-mon[22835]: [22/Apr/2024:22:06:13] ENGINE Serving on https://172.21.15.174:7150 2024-04-22T22:06:15.280 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:15 smithi192 ceph-mon[22835]: [22/Apr/2024:22:06:13] ENGINE Bus STARTED 2024-04-22T22:06:15.280 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:15 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T22:06:15.280 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:15 smithi192 ceph-mon[22835]: mgrmap e18: y(active, since 1.87725s), standbys: x 2024-04-22T22:06:15.280 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:15 smithi192 ceph-mon[22835]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi174=a;smithi192=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:06:15.280 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:15 smithi192 ceph-mon[22835]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:15.280 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:15 smithi192 ceph-mon[22835]: Saving service node-exporter spec with placement smithi174=a;smithi192=b;count:2 2024-04-22T22:06:15.280 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:15 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:15.280 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:15 smithi192 ceph-mon[22835]: Deploying daemon prometheus.a on smithi192 2024-04-22T22:06:15.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[21825]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:06:15.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[21825]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:06:15.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[21825]: [22/Apr/2024:22:06:13] ENGINE Bus STARTING 2024-04-22T22:06:15.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[21825]: [22/Apr/2024:22:06:13] ENGINE Serving on https://172.21.15.174:7150 2024-04-22T22:06:15.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[21825]: [22/Apr/2024:22:06:13] ENGINE Bus STARTED 2024-04-22T22:06:15.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T22:06:15.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[21825]: mgrmap e18: y(active, since 1.87725s), standbys: x 2024-04-22T22:06:15.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[21825]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi174=a;smithi192=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:06:15.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[21825]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:15.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[21825]: Saving service node-exporter spec with placement smithi174=a;smithi192=b;count:2 2024-04-22T22:06:15.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:15.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[21825]: Deploying daemon prometheus.a on smithi192 2024-04-22T22:06:15.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[26730]: Updating smithi174:/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:06:15.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[26730]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-04-22T22:06:15.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[26730]: [22/Apr/2024:22:06:13] ENGINE Bus STARTING 2024-04-22T22:06:15.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[26730]: [22/Apr/2024:22:06:13] ENGINE Serving on https://172.21.15.174:7150 2024-04-22T22:06:15.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[26730]: [22/Apr/2024:22:06:13] ENGINE Bus STARTED 2024-04-22T22:06:15.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-22T22:06:15.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[26730]: mgrmap e18: y(active, since 1.87725s), standbys: x 2024-04-22T22:06:15.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[26730]: from='client.34331 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi174=a;smithi192=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:06:15.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[26730]: pgmap v3: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:15.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[26730]: Saving service node-exporter spec with placement smithi174=a;smithi192=b;count:2 2024-04-22T22:06:15.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:15.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:15 smithi174 ceph-mon[26730]: Deploying daemon prometheus.a on smithi192 2024-04-22T22:06:16.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:16 smithi192 ceph-mon[22835]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:16.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:16 smithi192 ceph-mon[22835]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-22T22:06:16.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:16 smithi174 ceph-mon[21825]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:16.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:16 smithi174 ceph-mon[21825]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-22T22:06:16.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:16 smithi174 ceph-mon[26730]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:16.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:16 smithi174 ceph-mon[26730]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-22T22:06:17.026 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled alertmanager update... 2024-04-22T22:06:18.027 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:17 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:06:17] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:06:18.111 DEBUG:teuthology.orchestra.run.smithi174:alertmanager.a> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@alertmanager.a.service 2024-04-22T22:06:18.114 INFO:tasks.cephadm:Adding grafana.a on smithi192 2024-04-22T22:06:18.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph orch apply grafana '1;smithi192=a' 2024-04-22T22:06:18.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:18 smithi174 ceph-mon[21825]: from='client.34355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:06:18.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:18 smithi174 ceph-mon[21825]: Saving service alertmanager spec with placement smithi174=a;count:1 2024-04-22T22:06:18.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:18 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:18.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:18 smithi174 ceph-mon[21825]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:18.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:18 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:18.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:18 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:18.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:18 smithi174 ceph-mon[26730]: from='client.34355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:06:18.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:18 smithi174 ceph-mon[26730]: Saving service alertmanager spec with placement smithi174=a;count:1 2024-04-22T22:06:18.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:18 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:18.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:18 smithi174 ceph-mon[26730]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:18.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:18 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:18.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:18 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:18.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:18 smithi192 ceph-mon[22835]: from='client.34355 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi174=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:06:18.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:18 smithi192 ceph-mon[22835]: Saving service alertmanager spec with placement smithi174=a;count:1 2024-04-22T22:06:18.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:18 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:18.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:18 smithi192 ceph-mon[22835]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:18.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:18 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:18.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:18 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:20.447 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:20 smithi192 ceph-mon[22835]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:20.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:20 smithi174 ceph-mon[21825]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:20.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:20 smithi174 ceph-mon[26730]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:22.260 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:22 smithi192 ceph-mon[22835]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:22.366 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:22 smithi174 ceph-mon[21825]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:22.366 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:22 smithi174 ceph-mon[26730]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:22.573 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled grafana update... 2024-04-22T22:06:23.778 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:23 smithi174 ceph-mon[21825]: from='client.34361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:06:23.778 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:23 smithi174 ceph-mon[21825]: Saving service grafana spec with placement smithi192=a;count:1 2024-04-22T22:06:23.778 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:23 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:23.778 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:23 smithi174 ceph-mon[21825]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:23.779 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:23 smithi174 ceph-mon[26730]: from='client.34361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:06:23.779 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:23 smithi174 ceph-mon[26730]: Saving service grafana spec with placement smithi192=a;count:1 2024-04-22T22:06:23.779 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:23 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:23.779 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:23 smithi174 ceph-mon[26730]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:23.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:23 smithi192 ceph-mon[22835]: from='client.34361 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi192=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:06:23.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:23 smithi192 ceph-mon[22835]: Saving service grafana spec with placement smithi192=a;count:1 2024-04-22T22:06:23.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:23 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:23.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:23 smithi192 ceph-mon[22835]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:25.001 DEBUG:teuthology.orchestra.run.smithi192:grafana.a> sudo journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@grafana.a.service 2024-04-22T22:06:25.003 INFO:tasks.cephadm:Setting up client nodes... 2024-04-22T22:06:25.003 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T22:06:26.250 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-mon[22835]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:26.389 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:26 smithi174 ceph-mon[21825]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:26.389 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:26 smithi174 ceph-mon[26730]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:26.564 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 systemd[1]: Starting Ceph prometheus.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:06:26.893 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 podman[38820]: 2024-04-22 22:06:26.568442262 +0000 UTC m=+0.018798183 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-22T22:06:26.893 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 podman[38820]: 2024-04-22 22:06:26.673668051 +0000 UTC m=+0.124024005 container create 66f63bfb31b95157691f121fc8a29d2d3150887d63b049b494217fe958005abf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T22:06:26.893 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 podman[38820]: 2024-04-22 22:06:26.917690102 +0000 UTC m=+0.368046009 container init 66f63bfb31b95157691f121fc8a29d2d3150887d63b049b494217fe958005abf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T22:06:26.974 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:26 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:06:26] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:06:27.140 INFO:teuthology.orchestra.run.smithi174.stdout:[client.0] 2024-04-22T22:06:27.140 INFO:teuthology.orchestra.run.smithi174.stdout: key = AQBj3yZm1dV+CRAAvKHWx+5lV6Hj9aRli7MmKQ== 2024-04-22T22:06:27.200 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 podman[38820]: 2024-04-22 22:06:26.920402001 +0000 UTC m=+0.370757917 container start 66f63bfb31b95157691f121fc8a29d2d3150887d63b049b494217fe958005abf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T22:06:27.200 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.943Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-22T22:06:27.200 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.943Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-22T22:06:27.200 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.943Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-22T22:06:27.200 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.943Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi192 (none))" 2024-04-22T22:06:27.200 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.943Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-22T22:06:27.200 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.943Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-22T22:06:27.200 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.944Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-22T22:06:27.200 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.945Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-22T22:06:27.200 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.945Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-22T22:06:27.200 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.947Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-22T22:06:27.200 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.947Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.01µs 2024-04-22T22:06:27.200 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.947Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-22T22:06:27.201 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.948Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-22T22:06:27.201 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.948Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=34.931µs wal_replay_duration=397.595µs total_replay_duration=499.112µs 2024-04-22T22:06:27.201 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.948Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-22T22:06:27.201 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.948Z caller=main.go:947 level=info msg="TSDB started" 2024-04-22T22:06:27.201 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.948Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-22T22:06:27.201 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.964Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.258219ms db_storage=708ns remote_storage=1.327µs web_handler=229ns query_engine=873ns scrape=390.013µs scrape_sd=16.629µs notify=774ns notify_sd=1.324µs rules=14.690733ms 2024-04-22T22:06:27.201 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:06:26.964Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-22T22:06:27.201 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:27 smithi192 bash[38820]: 66f63bfb31b95157691f121fc8a29d2d3150887d63b049b494217fe958005abf 2024-04-22T22:06:27.201 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:06:27 smithi192 systemd[1]: Started Ceph prometheus.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:06:27.793 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.192:9095"}]: dispatch 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.192:9095"}]: dispatch 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:27.794 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2715817005' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[26730]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.795 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.192:9095"}]: dispatch 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.192:9095"}]: dispatch 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.796 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:06:27.797 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:27.797 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:27.797 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:27.797 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.797 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:06:27.797 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2715817005' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:06:27.797 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:06:27.797 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:27 smithi174 ceph-mon[21825]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T22:06:27.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:27.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:27.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.192:9095"}]: dispatch 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.192:9095"}]: dispatch 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:27.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:27.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:27.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:06:27.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2715817005' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:06:27.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:06:27.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:27 smithi192 ceph-mon[22835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T22:06:28.568 DEBUG:teuthology.orchestra.run.smithi174:> set -ex 2024-04-22T22:06:28.568 DEBUG:teuthology.orchestra.run.smithi174:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-22T22:06:28.568 DEBUG:teuthology.orchestra.run.smithi174:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-22T22:06:28.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-22T22:06:29.256 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:29 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:29.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:29 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:29.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:29 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:30.156 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:30 smithi174 ceph-mon[21825]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:30.156 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:30 smithi174 ceph-mon[26730]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:30.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:30 smithi192 ceph-mon[22835]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:32.271 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:32 smithi192 ceph-mon[22835]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:32.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:32 smithi174 ceph-mon[21825]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:32.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:32 smithi174 ceph-mon[26730]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:32.759 INFO:teuthology.orchestra.run.smithi192.stdout:[client.1] 2024-04-22T22:06:32.759 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQBo3yZm7Hh1LhAAf6ogFuqR579PIy73KWmfhw== 2024-04-22T22:06:33.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:33 smithi192 ceph-mon[22835]: from='client.? 172.21.15.192:0/3258406032' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:06:33.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:33 smithi192 ceph-mon[22835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:06:33.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:33 smithi192 ceph-mon[22835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T22:06:33.458 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-04-22T22:06:33.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-22T22:06:33.458 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-22T22:06:33.497 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-22T22:06:33.497 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-22T22:06:33.497 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-22T22:06:33.524 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:33 smithi174 ceph-mon[21825]: from='client.? 172.21.15.192:0/3258406032' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:06:33.524 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:33 smithi174 ceph-mon[21825]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:06:33.524 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:33 smithi174 ceph-mon[21825]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T22:06:33.524 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:33 smithi174 ceph-mon[26730]: from='client.? 172.21.15.192:0/3258406032' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:06:33.524 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:33 smithi174 ceph-mon[26730]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-22T22:06:33.525 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:33 smithi174 ceph-mon[26730]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-22T22:06:33.719 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:06:34.206 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:34 smithi192 ceph-mon[22835]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:34.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:34 smithi174 ceph-mon[21825]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:34.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:34 smithi174 ceph-mon[26730]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:34.867 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:34 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:06:34] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:06:35.139 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:06:35] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:06:35.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:35.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:35.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:35.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:35.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:35.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:35.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[21825]: Deploying daemon node-exporter.a on smithi174 2024-04-22T22:06:35.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[21825]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:35.748 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:35.749 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:35.749 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:35.749 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:35.749 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:35.749 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:35.749 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:35.749 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:35.749 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[26730]: Deploying daemon node-exporter.a on smithi174 2024-04-22T22:06:35.749 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[26730]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:35.749 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:35.749 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:35.758 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T22:06:36.065 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:35 smithi174 systemd[1]: Starting Ceph node-exporter.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:06:36.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:36.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:36.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:06:36.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:06:36.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:06:36.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:36.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:35 smithi192 ceph-mon[22835]: Deploying daemon node-exporter.a on smithi174 2024-04-22T22:06:36.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:35 smithi192 ceph-mon[22835]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:36.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:36.066 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:36.494 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":19,"active_gid":24349,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6800","nonce":1558366672},{"type":"v1","addr":"172.21.15.174:6801","nonce":1558366672}]},"active_addr":"172.21.15.174:6801/1558366672","active_change":"2024-04-22T22:06:12.184848+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34319,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.174:8443/","prometheus":"http://172.21.15.174:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":57,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":3356427004}]},{"addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":3021806114}]},{"addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":1404101500}]},{"addrvec":[{"type":"v2","addr":"172.21.15.174:0","nonce":1925482252}]}]}} 2024-04-22T22:06:36.499 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-22T22:06:36.500 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-22T22:06:36.500 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T22:06:36.772 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:06:36.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:36 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2853177546' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T22:06:36.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:36 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2853177546' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T22:06:36.815 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:36 smithi174 bash[42659]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-22T22:06:37.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:36 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2853177546' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-22T22:06:37.175 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:37 smithi174 bash[42659]: Getting image source signatures 2024-04-22T22:06:37.175 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:37 smithi174 bash[42659]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-22T22:06:37.175 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:37 smithi174 bash[42659]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-22T22:06:37.175 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:37 smithi174 bash[42659]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-22T22:06:38.297 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:38 smithi174 ceph-mon[21825]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:38.298 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:38 smithi174 ceph-mon[26730]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:38.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:38 smithi192 ceph-mon[22835]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:39.430 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T22:06:39.430 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":57,"fsid":"b7cbfdbc-00f3-11ef-bc93-c7b262605968","created":"2024-04-22T22:01:07.496253+0000","modified":"2024-04-22T22:06:12.184494+0000","last_up_change":"2024-04-22T22:05:54.368445+0000","last_in_change":"2024-04-22T22:05:36.482223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T22:04:05.963559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"10b65a80-27ba-4f98-b4a5-46610cbcee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2571929691},{"type":"v1","addr":"172.21.15.174:6803","nonce":2571929691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2571929691},{"type":"v1","addr":"172.21.15.174:6805","nonce":2571929691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2571929691},{"type":"v1","addr":"172.21.15.174:6809","nonce":2571929691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2571929691},{"type":"v1","addr":"172.21.15.174:6807","nonce":2571929691}]},"public_addr":"172.21.15.174:6803/2571929691","cluster_addr":"172.21.15.174:6805/2571929691","heartbeat_back_addr":"172.21.15.174:6809/2571929691","heartbeat_front_addr":"172.21.15.174:6807/2571929691","state":["exists","up"]},{"osd":1,"uuid":"b4d7e7fc-0a42-4a61-ba4f-a1693dc2a970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":222655237},{"type":"v1","addr":"172.21.15.174:6811","nonce":222655237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":222655237},{"type":"v1","addr":"172.21.15.174:6813","nonce":222655237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":222655237},{"type":"v1","addr":"172.21.15.174:6817","nonce":222655237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":222655237},{"type":"v1","addr":"172.21.15.174:6815","nonce":222655237}]},"public_addr":"172.21.15.174:6811/222655237","cluster_addr":"172.21.15.174:6813/222655237","heartbeat_back_addr":"172.21.15.174:6817/222655237","heartbeat_front_addr":"172.21.15.174:6815/222655237","state":["exists","up"]},{"osd":2,"uuid":"439d2d85-e8b4-4a98-bc6d-11fcab19c75d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":77924542},{"type":"v1","addr":"172.21.15.174:6819","nonce":77924542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":77924542},{"type":"v1","addr":"172.21.15.174:6821","nonce":77924542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":77924542},{"type":"v1","addr":"172.21.15.174:6825","nonce":77924542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":77924542},{"type":"v1","addr":"172.21.15.174:6823","nonce":77924542}]},"public_addr":"172.21.15.174:6819/77924542","cluster_addr":"172.21.15.174:6821/77924542","heartbeat_back_addr":"172.21.15.174:6825/77924542","heartbeat_front_addr":"172.21.15.174:6823/77924542","state":["exists","up"]},{"osd":3,"uuid":"d0ce25dc-969a-4435-ab08-96d78791c719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1588342953},{"type":"v1","addr":"172.21.15.174:6827","nonce":1588342953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1588342953},{"type":"v1","addr":"172.21.15.174:6829","nonce":1588342953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1588342953},{"type":"v1","addr":"172.21.15.174:6833","nonce":1588342953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1588342953},{"type":"v1","addr":"172.21.15.174:6831","nonce":1588342953}]},"public_addr":"172.21.15.174:6827/1588342953","cluster_addr":"172.21.15.174:6829/1588342953","heartbeat_back_addr":"172.21.15.174:6833/1588342953","heartbeat_front_addr":"172.21.15.174:6831/1588342953","state":["exists","up"]},{"osd":4,"uuid":"b95a43b7-c458-4f35-8494-52af4cc55af8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":469106314},{"type":"v1","addr":"172.21.15.192:6801","nonce":469106314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":469106314},{"type":"v1","addr":"172.21.15.192:6803","nonce":469106314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":469106314},{"type":"v1","addr":"172.21.15.192:6807","nonce":469106314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":469106314},{"type":"v1","addr":"172.21.15.192:6805","nonce":469106314}]},"public_addr":"172.21.15.192:6801/469106314","cluster_addr":"172.21.15.192:6803/469106314","heartbeat_back_addr":"172.21.15.192:6807/469106314","heartbeat_front_addr":"172.21.15.192:6805/469106314","state":["exists","up"]},{"osd":5,"uuid":"3400fe78-1500-44a8-a526-2a80969d5c54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":645383067},{"type":"v1","addr":"172.21.15.192:6809","nonce":645383067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":645383067},{"type":"v1","addr":"172.21.15.192:6811","nonce":645383067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":645383067},{"type":"v1","addr":"172.21.15.192:6815","nonce":645383067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":645383067},{"type":"v1","addr":"172.21.15.192:6813","nonce":645383067}]},"public_addr":"172.21.15.192:6809/645383067","cluster_addr":"172.21.15.192:6811/645383067","heartbeat_back_addr":"172.21.15.192:6815/645383067","heartbeat_front_addr":"172.21.15.192:6813/645383067","state":["exists","up"]},{"osd":6,"uuid":"f446ce85-1bc3-4f2c-a013-7841180ee701","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1564782047},{"type":"v1","addr":"172.21.15.192:6817","nonce":1564782047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1564782047},{"type":"v1","addr":"172.21.15.192:6819","nonce":1564782047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1564782047},{"type":"v1","addr":"172.21.15.192:6823","nonce":1564782047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1564782047},{"type":"v1","addr":"172.21.15.192:6821","nonce":1564782047}]},"public_addr":"172.21.15.192:6817/1564782047","cluster_addr":"172.21.15.192:6819/1564782047","heartbeat_back_addr":"172.21.15.192:6823/1564782047","heartbeat_front_addr":"172.21.15.192:6821/1564782047","state":["exists","up"]},{"osd":7,"uuid":"a033a707-0d56-4b05-8779-a85d5f337880","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":1603978747},{"type":"v1","addr":"172.21.15.192:6825","nonce":1603978747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":1603978747},{"type":"v1","addr":"172.21.15.192:6827","nonce":1603978747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":1603978747},{"type":"v1","addr":"172.21.15.192:6831","nonce":1603978747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":1603978747},{"type":"v1","addr":"172.21.15.192:6829","nonce":1603978747}]},"public_addr":"172.21.15.192:6825/1603978747","cluster_addr":"172.21.15.192:6827/1603978747","heartbeat_back_addr":"172.21.15.192:6831/1603978747","heartbeat_front_addr":"172.21.15.192:6829/1603978747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:03:16.803033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:03:40.119346+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:04:03.225265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:04:27.029259+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:04:49.376150+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:05:10.815050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:05:31.203539+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:05:52.478808+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.174:0/381710589":"2024-04-23T22:06:12.184447+0000","172.21.15.174:6801/3182264883":"2024-04-23T22:06:12.184447+0000","172.21.15.174:6800/3182264883":"2024-04-23T22:06:12.184447+0000","172.21.15.174:0/3592746357":"2024-04-23T22:01:53.599383+0000","172.21.15.174:0/1494365845":"2024-04-23T22:06:12.184447+0000","172.21.15.174:6801/2262747749":"2024-04-23T22:01:53.599383+0000","172.21.15.174:0/4157104731":"2024-04-23T22:01:53.599383+0000","172.21.15.174:0/3519428216":"2024-04-23T22:06:12.184447+0000","172.21.15.174:6800/2262747749":"2024-04-23T22:01:53.599383+0000","172.21.15.174:0/3967906832":"2024-04-23T22:01:53.599383+0000","172.21.15.174:0/1131845225":"2024-04-23T22:06:12.184447+0000","172.21.15.174:0/1922808348":"2024-04-23T22:01:31.026657+0000","172.21.15.174:0/2470543672":"2024-04-23T22:01:31.026657+0000","172.21.15.174:0/1468603511":"2024-04-23T22:01:31.026657+0000","172.21.15.174:6800/4261879208":"2024-04-23T22:01:31.026657+0000","172.21.15.174:6801/4261879208":"2024-04-23T22:01:31.026657+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T22:06:39.430 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:39 smithi174 bash[42659]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-22T22:06:39.814 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:39 smithi174 bash[42659]: Writing manifest to image destination 2024-04-22T22:06:40.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-mon[21825]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:40.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1000975691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:06:40.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-mon[26730]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:40.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1000975691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:06:40.315 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 podman[42659]: 2024-04-22 22:06:39.943659303 +0000 UTC m=+3.350020397 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-22T22:06:40.437 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-22T22:06:40.437 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-22T22:06:40.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:40 smithi192 ceph-mon[22835]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:40.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:40 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1000975691' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:06:40.675 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 podman[42659]: 2024-04-22 22:06:40.594497657 +0000 UTC m=+4.000858724 container create 5e1c68495a2c40c96d792f0c87600e32fa9f4c94efa21032c3330193029bc3f3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-22T22:06:40.887 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:06:41.014 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 podman[42659]: 2024-04-22 22:06:40.788680213 +0000 UTC m=+4.195041281 container init 5e1c68495a2c40c96d792f0c87600e32fa9f4c94efa21032c3330193029bc3f3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-22T22:06:41.014 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 podman[42659]: 2024-04-22 22:06:40.79175022 +0000 UTC m=+4.198111288 container start 5e1c68495a2c40c96d792f0c87600e32fa9f4c94efa21032c3330193029bc3f3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-22T22:06:41.014 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.795Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-22T22:06:41.014 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.795Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-22T22:06:41.014 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-22T22:06:41.014 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-22T22:06:41.014 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-22T22:06:41.015 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=arp 2024-04-22T22:06:41.015 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-22T22:06:41.015 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-22T22:06:41.015 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-22T22:06:41.015 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-22T22:06:41.015 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-22T22:06:41.015 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-22T22:06:41.015 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-22T22:06:41.015 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-22T22:06:41.015 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=edac 2024-04-22T22:06:41.016 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-22T22:06:41.016 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-22T22:06:41.016 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-22T22:06:41.016 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-22T22:06:41.016 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-22T22:06:41.016 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-22T22:06:41.016 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-22T22:06:41.016 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-22T22:06:41.016 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-22T22:06:41.016 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-22T22:06:41.016 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-22T22:06:41.016 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-22T22:06:41.016 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=os 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=stat 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=time 2024-04-22T22:06:41.017 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-22T22:06:41.018 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=uname 2024-04-22T22:06:41.018 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-22T22:06:41.018 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-22T22:06:41.018 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-22T22:06:41.018 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.796Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-22T22:06:41.018 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a[42863]: ts=2024-04-22T22:06:40.797Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-22T22:06:41.018 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 bash[42659]: 5e1c68495a2c40c96d792f0c87600e32fa9f4c94efa21032c3330193029bc3f3 2024-04-22T22:06:41.018 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:06:40 smithi174 systemd[1]: Started Ceph node-exporter.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:06:41.773 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:41 smithi192 systemd[1]: Starting Ceph node-exporter.b for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:06:42.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:41 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:42.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:41 smithi192 ceph-mon[22835]: Deploying daemon node-exporter.b on smithi192 2024-04-22T22:06:42.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:41 smithi192 ceph-mon[22835]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:42.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:41 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:42.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:41 smithi174 ceph-mon[21825]: Deploying daemon node-exporter.b on smithi192 2024-04-22T22:06:42.066 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:41 smithi174 ceph-mon[21825]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:42.066 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:41 smithi192 bash[39780]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-22T22:06:42.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:41 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:42.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:41 smithi174 ceph-mon[26730]: Deploying daemon node-exporter.b on smithi192 2024-04-22T22:06:42.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:41 smithi174 ceph-mon[26730]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:42.565 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:42 smithi192 bash[39780]: Getting image source signatures 2024-04-22T22:06:42.565 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:42 smithi192 bash[39780]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-22T22:06:42.565 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:42 smithi192 bash[39780]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-22T22:06:42.565 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:42 smithi192 bash[39780]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-22T22:06:42.740 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T22:06:42.741 INFO:teuthology.orchestra.run.smithi174.stdout:{"epoch":57,"fsid":"b7cbfdbc-00f3-11ef-bc93-c7b262605968","created":"2024-04-22T22:01:07.496253+0000","modified":"2024-04-22T22:06:12.184494+0000","last_up_change":"2024-04-22T22:05:54.368445+0000","last_in_change":"2024-04-22T22:05:36.482223+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-22T22:04:05.963559+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"10b65a80-27ba-4f98-b4a5-46610cbcee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6802","nonce":2571929691},{"type":"v1","addr":"172.21.15.174:6803","nonce":2571929691}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6804","nonce":2571929691},{"type":"v1","addr":"172.21.15.174:6805","nonce":2571929691}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6808","nonce":2571929691},{"type":"v1","addr":"172.21.15.174:6809","nonce":2571929691}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6806","nonce":2571929691},{"type":"v1","addr":"172.21.15.174:6807","nonce":2571929691}]},"public_addr":"172.21.15.174:6803/2571929691","cluster_addr":"172.21.15.174:6805/2571929691","heartbeat_back_addr":"172.21.15.174:6809/2571929691","heartbeat_front_addr":"172.21.15.174:6807/2571929691","state":["exists","up"]},{"osd":1,"uuid":"b4d7e7fc-0a42-4a61-ba4f-a1693dc2a970","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6810","nonce":222655237},{"type":"v1","addr":"172.21.15.174:6811","nonce":222655237}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6812","nonce":222655237},{"type":"v1","addr":"172.21.15.174:6813","nonce":222655237}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6816","nonce":222655237},{"type":"v1","addr":"172.21.15.174:6817","nonce":222655237}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6814","nonce":222655237},{"type":"v1","addr":"172.21.15.174:6815","nonce":222655237}]},"public_addr":"172.21.15.174:6811/222655237","cluster_addr":"172.21.15.174:6813/222655237","heartbeat_back_addr":"172.21.15.174:6817/222655237","heartbeat_front_addr":"172.21.15.174:6815/222655237","state":["exists","up"]},{"osd":2,"uuid":"439d2d85-e8b4-4a98-bc6d-11fcab19c75d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6818","nonce":77924542},{"type":"v1","addr":"172.21.15.174:6819","nonce":77924542}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6820","nonce":77924542},{"type":"v1","addr":"172.21.15.174:6821","nonce":77924542}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6824","nonce":77924542},{"type":"v1","addr":"172.21.15.174:6825","nonce":77924542}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6822","nonce":77924542},{"type":"v1","addr":"172.21.15.174:6823","nonce":77924542}]},"public_addr":"172.21.15.174:6819/77924542","cluster_addr":"172.21.15.174:6821/77924542","heartbeat_back_addr":"172.21.15.174:6825/77924542","heartbeat_front_addr":"172.21.15.174:6823/77924542","state":["exists","up"]},{"osd":3,"uuid":"d0ce25dc-969a-4435-ab08-96d78791c719","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6826","nonce":1588342953},{"type":"v1","addr":"172.21.15.174:6827","nonce":1588342953}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6828","nonce":1588342953},{"type":"v1","addr":"172.21.15.174:6829","nonce":1588342953}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6832","nonce":1588342953},{"type":"v1","addr":"172.21.15.174:6833","nonce":1588342953}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.174:6830","nonce":1588342953},{"type":"v1","addr":"172.21.15.174:6831","nonce":1588342953}]},"public_addr":"172.21.15.174:6827/1588342953","cluster_addr":"172.21.15.174:6829/1588342953","heartbeat_back_addr":"172.21.15.174:6833/1588342953","heartbeat_front_addr":"172.21.15.174:6831/1588342953","state":["exists","up"]},{"osd":4,"uuid":"b95a43b7-c458-4f35-8494-52af4cc55af8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":469106314},{"type":"v1","addr":"172.21.15.192:6801","nonce":469106314}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":469106314},{"type":"v1","addr":"172.21.15.192:6803","nonce":469106314}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":469106314},{"type":"v1","addr":"172.21.15.192:6807","nonce":469106314}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":469106314},{"type":"v1","addr":"172.21.15.192:6805","nonce":469106314}]},"public_addr":"172.21.15.192:6801/469106314","cluster_addr":"172.21.15.192:6803/469106314","heartbeat_back_addr":"172.21.15.192:6807/469106314","heartbeat_front_addr":"172.21.15.192:6805/469106314","state":["exists","up"]},{"osd":5,"uuid":"3400fe78-1500-44a8-a526-2a80969d5c54","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":645383067},{"type":"v1","addr":"172.21.15.192:6809","nonce":645383067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":645383067},{"type":"v1","addr":"172.21.15.192:6811","nonce":645383067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":645383067},{"type":"v1","addr":"172.21.15.192:6815","nonce":645383067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":645383067},{"type":"v1","addr":"172.21.15.192:6813","nonce":645383067}]},"public_addr":"172.21.15.192:6809/645383067","cluster_addr":"172.21.15.192:6811/645383067","heartbeat_back_addr":"172.21.15.192:6815/645383067","heartbeat_front_addr":"172.21.15.192:6813/645383067","state":["exists","up"]},{"osd":6,"uuid":"f446ce85-1bc3-4f2c-a013-7841180ee701","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":44,"up_thru":47,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":1564782047},{"type":"v1","addr":"172.21.15.192:6817","nonce":1564782047}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":1564782047},{"type":"v1","addr":"172.21.15.192:6819","nonce":1564782047}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":1564782047},{"type":"v1","addr":"172.21.15.192:6823","nonce":1564782047}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":1564782047},{"type":"v1","addr":"172.21.15.192:6821","nonce":1564782047}]},"public_addr":"172.21.15.192:6817/1564782047","cluster_addr":"172.21.15.192:6819/1564782047","heartbeat_back_addr":"172.21.15.192:6823/1564782047","heartbeat_front_addr":"172.21.15.192:6821/1564782047","state":["exists","up"]},{"osd":7,"uuid":"a033a707-0d56-4b05-8779-a85d5f337880","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":1603978747},{"type":"v1","addr":"172.21.15.192:6825","nonce":1603978747}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":1603978747},{"type":"v1","addr":"172.21.15.192:6827","nonce":1603978747}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":1603978747},{"type":"v1","addr":"172.21.15.192:6831","nonce":1603978747}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":1603978747},{"type":"v1","addr":"172.21.15.192:6829","nonce":1603978747}]},"public_addr":"172.21.15.192:6825/1603978747","cluster_addr":"172.21.15.192:6827/1603978747","heartbeat_back_addr":"172.21.15.192:6831/1603978747","heartbeat_front_addr":"172.21.15.192:6829/1603978747","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:03:16.803033+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:03:40.119346+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:04:03.225265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:04:27.029259+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:04:49.376150+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:05:10.815050+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:05:31.203539+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-22T22:05:52.478808+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.174:0/381710589":"2024-04-23T22:06:12.184447+0000","172.21.15.174:6801/3182264883":"2024-04-23T22:06:12.184447+0000","172.21.15.174:6800/3182264883":"2024-04-23T22:06:12.184447+0000","172.21.15.174:0/3592746357":"2024-04-23T22:01:53.599383+0000","172.21.15.174:0/1494365845":"2024-04-23T22:06:12.184447+0000","172.21.15.174:6801/2262747749":"2024-04-23T22:01:53.599383+0000","172.21.15.174:0/4157104731":"2024-04-23T22:01:53.599383+0000","172.21.15.174:0/3519428216":"2024-04-23T22:06:12.184447+0000","172.21.15.174:6800/2262747749":"2024-04-23T22:01:53.599383+0000","172.21.15.174:0/3967906832":"2024-04-23T22:01:53.599383+0000","172.21.15.174:0/1131845225":"2024-04-23T22:06:12.184447+0000","172.21.15.174:0/1922808348":"2024-04-23T22:01:31.026657+0000","172.21.15.174:0/2470543672":"2024-04-23T22:01:31.026657+0000","172.21.15.174:0/1468603511":"2024-04-23T22:01:31.026657+0000","172.21.15.174:6800/4261879208":"2024-04-23T22:01:31.026657+0000","172.21.15.174:6801/4261879208":"2024-04-23T22:01:31.026657+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-22T22:06:43.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3146969145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:06:43.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:43 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3146969145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:06:43.315 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 bash[39780]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-22T22:06:43.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:43 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3146969145' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-22T22:06:43.443 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-22T22:06:43.443 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-22T22:06:43.443 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-22T22:06:43.444 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-22T22:06:43.444 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-22T22:06:43.445 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-22T22:06:43.445 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-22T22:06:43.445 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-22T22:06:43.694 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 bash[39780]: Writing manifest to image destination 2024-04-22T22:06:43.694 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 podman[39780]: 2024-04-22 22:06:43.595699308 +0000 UTC m=+1.891184086 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-22T22:06:43.883 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:06:43.883 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:06:43.883 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:06:43.887 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:06:43.887 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:06:43.888 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:06:43.890 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:06:43.891 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:06:43.996 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 podman[39780]: 2024-04-22 22:06:43.719398878 +0000 UTC m=+2.014883642 container create 71966acd94b16a39818e38cf5a6d06c8c1475c162048cad096241c4f9a708790 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-22T22:06:43.996 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 podman[39780]: 2024-04-22 22:06:43.904742005 +0000 UTC m=+2.200226770 container init 71966acd94b16a39818e38cf5a6d06c8c1475c162048cad096241c4f9a708790 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-22T22:06:43.996 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 podman[39780]: 2024-04-22 22:06:43.907311056 +0000 UTC m=+2.202795821 container start 71966acd94b16a39818e38cf5a6d06c8c1475c162048cad096241c4f9a708790 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-22T22:06:43.996 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.911Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-22T22:06:43.996 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.911Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-22T22:06:43.997 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.911Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-22T22:06:43.997 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.911Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-22T22:06:43.997 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-22T22:06:43.997 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=arp 2024-04-22T22:06:43.997 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-22T22:06:43.997 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-22T22:06:43.997 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-22T22:06:43.997 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-22T22:06:43.997 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-22T22:06:43.997 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-22T22:06:43.997 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-22T22:06:43.998 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-22T22:06:43.998 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=edac 2024-04-22T22:06:43.998 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-22T22:06:43.998 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-22T22:06:43.998 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-22T22:06:43.998 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-22T22:06:43.998 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-22T22:06:43.998 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-22T22:06:43.998 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-22T22:06:43.998 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-22T22:06:43.998 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=os 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-22T22:06:43.999 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=stat 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=time 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=uname 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.912Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b[39830]: ts=2024-04-22T22:06:43.913Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 bash[39780]: 71966acd94b16a39818e38cf5a6d06c8c1475c162048cad096241c4f9a708790 2024-04-22T22:06:44.000 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:06:43 smithi192 systemd[1]: Started Ceph node-exporter.b for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:06:44.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:44 smithi174 ceph-mon[21825]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:44.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:44.278 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:44 smithi174 ceph-mon[26730]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:44.278 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:44.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:44 smithi192 ceph-mon[22835]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:44.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:45.192 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:45 smithi192 ceph-mon[22835]: Deploying daemon alertmanager.a on smithi174 2024-04-22T22:06:45.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:45 smithi174 ceph-mon[21825]: Deploying daemon alertmanager.a on smithi174 2024-04-22T22:06:45.343 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:45 smithi174 ceph-mon[26730]: Deploying daemon alertmanager.a on smithi174 2024-04-22T22:06:46.313 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:46 smithi174 ceph-mon[21825]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:46.313 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:46 smithi174 ceph-mon[26730]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:46 smithi192 ceph-mon[22835]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:46.968 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:46 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:06:46] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:06:47.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:47 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:06:47] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:06:47.857 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:47 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:47.857 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:47 smithi174 ceph-mon[21825]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:47.857 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:47 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:47.857 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:47 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:47.857 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:47 smithi174 ceph-mon[26730]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:47.857 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:47 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:48.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:47 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:48.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:47 smithi192 ceph-mon[22835]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:48.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:47 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:49.065 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:06:48 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:06:48] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:06:49.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:49 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:49.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:49 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:49.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:49 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:49.504 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:49 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:49.504 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:49 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:49.504 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:49 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:49.504 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:49 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:49.504 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:49 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:49.504 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:49 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:06:50.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:50 smithi174 ceph-mon[21825]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:50.384 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:50 smithi174 ceph-mon[26730]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:50.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:50 smithi192 ceph-mon[22835]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:52.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:52 smithi174 ceph-mon[21825]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:52.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:52 smithi174 ceph-mon[26730]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:52.369 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:52 smithi192 ceph-mon[22835]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:54.468 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:54 smithi174 ceph-mon[21825]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:54.468 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:54 smithi174 ceph-mon[26730]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:54.507 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:54 smithi192 ceph-mon[22835]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:55.039 INFO:teuthology.orchestra.run.smithi174.stdout:77309411363 2024-04-22T22:06:55.040 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-22T22:06:56.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:56 smithi192 ceph-mon[22835]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:56.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:56 smithi174 ceph-mon[21825]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:56.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:56 smithi174 ceph-mon[26730]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:58.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:06:58 smithi174 ceph-mon[21825]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:58.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:06:58 smithi174 ceph-mon[26730]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:06:58.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:06:58 smithi192 ceph-mon[22835]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:00.156 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:00 smithi174 ceph-mon[21825]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:00.157 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:00 smithi174 ceph-mon[26730]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:00.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:00 smithi192 ceph-mon[22835]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:02.219 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:02 smithi174 ceph-mon[21825]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:02.220 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:02 smithi174 ceph-mon[26730]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:02.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:02 smithi192 ceph-mon[22835]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:02.889 INFO:teuthology.orchestra.run.smithi174.stdout:128849018908 2024-04-22T22:07:02.889 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-22T22:07:03.676 INFO:teuthology.orchestra.run.smithi174.stdout:107374182433 2024-04-22T22:07:03.677 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-22T22:07:03.982 INFO:teuthology.orchestra.run.smithi174.stdout:223338299408 2024-04-22T22:07:03.982 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-22T22:07:04.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:04 smithi192 ceph-mon[22835]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:04.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:04 smithi174 ceph-mon[26730]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:04.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:04 smithi174 ceph-mon[21825]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:04.725 INFO:teuthology.orchestra.run.smithi174.stdout:55834574890 2024-04-22T22:07:04.726 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-22T22:07:05.852 INFO:teuthology.orchestra.run.smithi174.stdout:188978561043 2024-04-22T22:07:05.852 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-22T22:07:06.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:06 smithi192 ceph-mon[22835]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:06.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:06 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:06.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:06 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:06.484 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:06 smithi174 ceph-mon[21825]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:06.484 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:06 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:06.484 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:06 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:06.484 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:06 smithi174 ceph-mon[26730]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:06.484 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:06 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:06.485 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:06 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:06.485 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:07:06 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:07:06] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:07:06.506 INFO:teuthology.orchestra.run.smithi174.stdout:34359738416 2024-04-22T22:07:06.506 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-22T22:07:06.684 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:07:06.710 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:07:06.710 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:07:06.711 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:07:06.711 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:07:07.175 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:07:06 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:07:06] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:07:08.045 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:07 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:08.045 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:07 smithi174 ceph-mon[21825]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:08.045 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:07 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:08.045 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:07 smithi174 ceph-mon[26730]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:08.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:07 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:08.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:07 smithi192 ceph-mon[22835]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:09.476 INFO:teuthology.orchestra.run.smithi174.stdout:158913789975 2024-04-22T22:07:09.477 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-22T22:07:09.990 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:07:10.278 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:10 smithi174 ceph-mon[21825]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:10.278 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:10 smithi174 ceph-mon[26730]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:10.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:10 smithi192 ceph-mon[22835]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:10.715 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:07:11.216 INFO:teuthology.orchestra.run.smithi174.stdout:107374182435 2024-04-22T22:07:12.290 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:12 smithi174 ceph-mon[21825]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:12.290 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:12 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2719537233' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T22:07:12.290 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:12 smithi174 ceph-mon[26730]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:12.290 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:12 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2719537233' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T22:07:12.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:12 smithi192 ceph-mon[22835]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:12.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:12 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2719537233' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-22T22:07:12.620 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182433 got 107374182435 for osd.3 2024-04-22T22:07:12.621 DEBUG:teuthology.parallel:result is None 2024-04-22T22:07:13.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:07:13.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:07:13.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:07:14.391 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:14 smithi174 ceph-mon[26730]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:14.391 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:07:14.391 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:14 smithi174 ceph-mon[21825]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:14.391 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:07:14.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:14 smithi192 ceph-mon[22835]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:14.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:07:15.109 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:07:16.211 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:16 smithi174 ceph-mon[21825]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:16.211 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:16 smithi174 ceph-mon[26730]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:16.396 INFO:teuthology.orchestra.run.smithi174.stdout:188978561046 2024-04-22T22:07:16.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:16 smithi192 ceph-mon[22835]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:17.065 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:17 smithi174 podman[45271]: 2024-04-22 22:07:17.041757813 +0000 UTC m=+2.224958113 container create ca09846bf6a44718bc5bb12b4175738541a9ee1718fe483eb144f2000b63eb85 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:17.286 INFO:teuthology.orchestra.run.smithi174.stdout:77309411368 2024-04-22T22:07:17.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:17 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1504872647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T22:07:17.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:17 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3859227733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T22:07:17.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:17 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1504872647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T22:07:17.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:17 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3859227733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T22:07:17.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:17 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1504872647' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-22T22:07:17.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:17 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3859227733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-22T22:07:18.014 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411363 got 77309411368 for osd.2 2024-04-22T22:07:18.014 DEBUG:teuthology.parallel:result is None 2024-04-22T22:07:18.267 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:18 smithi174 ceph-mon[21825]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:18.267 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:18 smithi174 ceph-mon[26730]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:18.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:18 smithi192 ceph-mon[22835]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:19.612 INFO:tasks.cephadm.ceph_manager.ceph:need seq 188978561043 got 188978561046 for osd.6 2024-04-22T22:07:19.612 DEBUG:teuthology.parallel:result is None 2024-04-22T22:07:20.186 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:20 smithi174 ceph-mon[21825]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:20.187 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:20 smithi174 ceph-mon[26730]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:20.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:20 smithi192 ceph-mon[22835]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:20.711 INFO:teuthology.orchestra.run.smithi174.stdout:158913789978 2024-04-22T22:07:20.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:20 smithi174 podman[45271]: 2024-04-22 22:07:20.757777851 +0000 UTC m=+5.940978154 container init ca09846bf6a44718bc5bb12b4175738541a9ee1718fe483eb144f2000b63eb85 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:20.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:20 smithi174 podman[45271]: 2024-04-22 22:07:20.762256687 +0000 UTC m=+5.945456990 container start ca09846bf6a44718bc5bb12b4175738541a9ee1718fe483eb144f2000b63eb85 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:20.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:20 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[45427]: level=info ts=2024-04-22T22:07:20.772Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-22T22:07:20.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:20 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[45427]: level=info ts=2024-04-22T22:07:20.772Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-22T22:07:20.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:20 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[45427]: level=info ts=2024-04-22T22:07:20.774Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.174 port=9094 2024-04-22T22:07:20.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:20 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[45427]: level=info ts=2024-04-22T22:07:20.775Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-22T22:07:20.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:20 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[45427]: level=info ts=2024-04-22T22:07:20.799Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-22T22:07:20.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:20 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[45427]: level=info ts=2024-04-22T22:07:20.800Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-22T22:07:20.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:20 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[45427]: level=info ts=2024-04-22T22:07:20.802Z caller=main.go:518 msg=Listening address=:9093 2024-04-22T22:07:20.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:20 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[45427]: level=info ts=2024-04-22T22:07:20.802Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-22T22:07:21.187 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:21 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3728675640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T22:07:21.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:21 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3728675640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T22:07:21.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:21 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3728675640' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-22T22:07:22.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:22 smithi192 ceph-mon[22835]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:22.469 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:22 smithi174 ceph-mon[21825]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:22 smithi174 ceph-mon[26730]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:22.987 INFO:teuthology.orchestra.run.smithi174.stdout:128849018912 2024-04-22T22:07:23.001 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:22 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[45427]: level=info ts=2024-04-22T22:07:22.775Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000325757s 2024-04-22T22:07:23.233 INFO:teuthology.orchestra.run.smithi174.stdout:223338299412 2024-04-22T22:07:23.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:23 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1822971488' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T22:07:23.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:23 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1822971488' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T22:07:23.498 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:23 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1822971488' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-22T22:07:23.892 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299408 got 223338299412 for osd.7 2024-04-22T22:07:23.893 DEBUG:teuthology.parallel:result is None 2024-04-22T22:07:24.386 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:24 smithi174 ceph-mon[21825]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:24.386 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:24 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3804902389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T22:07:24.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:24 smithi174 ceph-mon[26730]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:24.387 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:24 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3804902389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T22:07:24.387 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018908 got 128849018912 for osd.4 2024-04-22T22:07:24.388 DEBUG:teuthology.parallel:result is None 2024-04-22T22:07:24.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:24 smithi192 ceph-mon[22835]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:24.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:24 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3804902389' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-22T22:07:24.667 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:24 smithi174 bash[45271]: ca09846bf6a44718bc5bb12b4175738541a9ee1718fe483eb144f2000b63eb85 2024-04-22T22:07:24.667 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:24 smithi174 systemd[1]: Started Ceph alertmanager.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:07:24.760 INFO:teuthology.orchestra.run.smithi174.stdout:55834574894 2024-04-22T22:07:25.985 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:25.986 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:25.986 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:25.986 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T22:07:25.986 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[21825]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T22:07:25.986 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:25.986 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[21825]: Deploying daemon grafana.a on smithi192 2024-04-22T22:07:25.986 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3397883652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T22:07:25.986 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[21825]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:25.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:25.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:25.986 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:25.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T22:07:25.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[26730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T22:07:25.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:25.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[26730]: Deploying daemon grafana.a on smithi192 2024-04-22T22:07:25.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3397883652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T22:07:25.987 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:25 smithi174 ceph-mon[26730]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:26.008 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789975 got 158913789978 for osd.5 2024-04-22T22:07:26.009 DEBUG:teuthology.parallel:result is None 2024-04-22T22:07:26.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:25 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:26.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:25 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:26.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:25 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:26.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:25 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T22:07:26.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:25 smithi192 ceph-mon[22835]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-22T22:07:26.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:25 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:26.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:25 smithi192 ceph-mon[22835]: Deploying daemon grafana.a on smithi192 2024-04-22T22:07:26.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:25 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3397883652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-22T22:07:26.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:25 smithi192 ceph-mon[22835]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:26.248 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:07:26 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:07:26] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:07:26.963 INFO:teuthology.orchestra.run.smithi174.stdout:34359738420 2024-04-22T22:07:27.200 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:07:26 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:07:26] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:07:27.201 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:27.201 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:27.201 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:27 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:27.201 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:27 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/821152864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T22:07:27.201 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:27 smithi174 ceph-mon[21825]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:27.201 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:27.201 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:27.202 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:27 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:27.202 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:27 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/821152864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T22:07:27.202 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:27 smithi174 ceph-mon[26730]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:27.202 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574890 got 55834574894 for osd.1 2024-04-22T22:07:27.203 DEBUG:teuthology.parallel:result is None 2024-04-22T22:07:27.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:27.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:27.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:27 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:27.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:27 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/821152864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-22T22:07:27.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:27 smithi192 ceph-mon[22835]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:27.804 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738420 for osd.0 2024-04-22T22:07:27.805 DEBUG:teuthology.parallel:result is None 2024-04-22T22:07:27.805 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-22T22:07:27.805 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T22:07:28.770 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:07:28 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:07:28] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:07:28.775 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:07:29.455 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:29 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:29.455 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:29 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:29.455 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:29 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:29.455 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:29 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:29.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:29 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:29.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:29 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:30.294 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:30 smithi174 ceph-mon[21825]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:30.294 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:30 smithi174 ceph-mon[26730]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:30.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:30 smithi192 ceph-mon[22835]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:30.965 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:30 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[45427]: level=info ts=2024-04-22T22:07:30.777Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002425625s 2024-04-22T22:07:31.573 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T22:07:31.574 INFO:teuthology.orchestra.run.smithi174.stderr:dumped all 2024-04-22T22:07:32.351 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:32 smithi174 ceph-mon[21825]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:32.351 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:32 smithi174 ceph-mon[26730]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:32.373 INFO:teuthology.orchestra.run.smithi174.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-04-22T22:07:31.073237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48500,"kb_used_data":5940,"kb_used_omap":0,"kb_used_meta":42496,"kb_avail":749912716,"statfs":{"total":767960285184,"available":767910621184,"internally_reserved":0,"allocated":6082560,"data_stored":3418927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43515904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002596"},"pg_stats":[{"pgid":"1.0","version":"57'128","reported_seq":266,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T22:06:13.146354+0000","last_change":"2024-04-22T22:05:58.052913+0000","last_active":"2024-04-22T22:06:13.146354+0000","last_peered":"2024-04-22T22:06:13.146354+0000","last_clean":"2024-04-22T22:06:13.146354+0000","last_became_active":"2024-04-22T22:05:58.052526+0000","last_became_peered":"2024-04-22T22:05:58.052526+0000","last_unstale":"2024-04-22T22:06:13.146354+0000","last_undegraded":"2024-04-22T22:06:13.146354+0000","last_fullsized":"2024-04-22T22:06:13.146354+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T22:04:06.181302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T22:04:06.181302+0000","last_clean_scrub_stamp":"2024-04-22T22:04:06.181302+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:14:18.226886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299414,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6068,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739084,"statfs":{"total":95995035648,"available":95988822016,"internally_reserved":0,"allocated":1093632,"data_stored":757979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.375}]},{"osd":1,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.47099999999999997}]},{"osd":2,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.38800000000000001}]},{"osd":3,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.435}]},{"osd":4,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.42199999999999999}]},{"osd":5,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.45300000000000001}]},{"osd":6,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.442}]}]},{"osd":6,"up_from":44,"seq":188978561049,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6132,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739020,"statfs":{"total":95995035648,"available":95988756480,"internally_reserved":0,"allocated":1093632,"data_stored":757979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Apr 22 22:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.46000000000000002}]},{"osd":2,"last update":"Mon Apr 22 22:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.51700000000000002}]},{"osd":3,"last update":"Mon Apr 22 22:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.35599999999999998}]},{"osd":4,"last update":"Mon Apr 22 22:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.44},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.34499999999999997}]},{"osd":5,"last update":"Mon Apr 22 22:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.42899999999999999}]},{"osd":7,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.5},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.56599999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6252,"kb_used_data":548,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738900,"statfs":{"total":95995035648,"available":95988633600,"internally_reserved":0,"allocated":561152,"data_stored":229051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.36699999999999999,"15min":0.34799999999999998},"min":{"1min":0.23499999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.434,"5min":0.376,"15min":0.35199999999999998},"min":{"1min":0.28000000000000003,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.36699999999999999}]},{"osd":2,"last update":"Mon Apr 22 22:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.38500000000000001,"15min":0.373},"min":{"1min":0.25700000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.40799999999999997,"15min":0.40000000000000002},"min":{"1min":0.27500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.31}]},{"osd":3,"last update":"Mon Apr 22 22:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.379},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.34300000000000003}]},{"osd":4,"last update":"Mon Apr 22 22:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42899999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61399999999999999,"15min":0.61599999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78300000000000003,"5min":0.872,"15min":0.872},"last":0.39100000000000001}]},{"osd":5,"last update":"Mon Apr 22 22:07:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61099999999999999,"15min":0.61899999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.59099999999999997,"15min":0.59499999999999997},"min":{"1min":0.47399999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.754,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.40400000000000003}]},{"osd":6,"last update":"Mon Apr 22 22:06:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.51100000000000001}]},{"osd":7,"last update":"Mon Apr 22 22:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.434},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.439}]}]},{"osd":0,"up_from":8,"seq":34359738421,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6900,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738252,"statfs":{"total":95995035648,"available":95987970048,"internally_reserved":0,"allocated":1093632,"data_stored":757979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 22:06:51 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.35699999999999998,"15min":0.34699999999999998},"min":{"1min":0.26300000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.47699999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36899999999999999,"15min":0.35499999999999998},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.442}]},{"osd":2,"last update":"Mon Apr 22 22:07:09 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41499999999999998,"15min":0.42199999999999999},"min":{"1min":0.246,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.48599999999999999,"5min":0.62,"15min":0.62},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.41499999999999998,"15min":0.42299999999999999},"min":{"1min":0.252,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.48499999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.35799999999999998}]},{"osd":3,"last update":"Mon Apr 22 22:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45500000000000002,"15min":0.45900000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.45000000000000001,"15min":0.45400000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.41799999999999998}]},{"osd":4,"last update":"Mon Apr 22 22:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57899999999999996,"15min":0.58199999999999996},"min":{"1min":0.40000000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.81699999999999995,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.48799999999999999}]},{"osd":5,"last update":"Mon Apr 22 22:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.497},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.41899999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79300000000000004,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.504}]},{"osd":6,"last update":"Mon Apr 22 22:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.53100000000000003}]},{"osd":7,"last update":"Mon Apr 22 22:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.57499999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411371,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6188,"kb_used_data":548,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":561152,"data_stored":229051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38800000000000001,"15min":0.36299999999999999},"min":{"1min":0.28199999999999997,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.51800000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.40100000000000002,"15min":0.38700000000000001},"min":{"1min":0.29399999999999998,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.34399999999999997}]},{"osd":1,"last update":"Mon Apr 22 22:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.41099999999999998,"15min":0.39700000000000002},"min":{"1min":0.309,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.58599999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.39800000000000002,"15min":0.38100000000000001},"min":{"1min":0.29899999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.41299999999999998}]},{"osd":3,"last update":"Mon Apr 22 22:06:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47099999999999997,"15min":0.47199999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Apr 22 22:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.44700000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.42499999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77400000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.46200000000000002}]},{"osd":5,"last update":"Mon Apr 22 22:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59299999999999997,"15min":0.58999999999999997},"min":{"1min":0.40300000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81000000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.52800000000000002}]},{"osd":6,"last update":"Mon Apr 22 22:06:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.441}]},{"osd":7,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.53800000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5676,"kb_used_data":548,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739476,"statfs":{"total":95995035648,"available":95989223424,"internally_reserved":0,"allocated":561152,"data_stored":229051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.40000000000000002,"15min":0.40600000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.61199999999999999,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60099999999999998,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.45900000000000002}]},{"osd":1,"last update":"Mon Apr 22 22:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58299999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.34999999999999998}]},{"osd":2,"last update":"Mon Apr 22 22:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.441,"15min":0.443},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.61699999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.52400000000000002}]},{"osd":4,"last update":"Mon Apr 22 22:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.435},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.41099999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.58999999999999997}]},{"osd":5,"last update":"Mon Apr 22 22:07:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.42799999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73999999999999999,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.42499999999999999}]},{"osd":6,"last update":"Mon Apr 22 22:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.5},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Apr 22 22:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.53700000000000003}]}]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5676,"kb_used_data":548,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739476,"statfs":{"total":95995035648,"available":95989223424,"internally_reserved":0,"allocated":561152,"data_stored":229051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:07:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73299999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40000000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Apr 22 22:07:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54700000000000004,"15min":0.55200000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.38},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.36499999999999999}]},{"osd":2,"last update":"Mon Apr 22 22:07:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.39500000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.434},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54500000000000004,"15min":0.55100000000000005},"min":{"1min":0.377,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.42199999999999999}]},{"osd":3,"last update":"Mon Apr 22 22:07:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54200000000000004,"15min":0.54800000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Apr 22 22:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40300000000000002,"15min":0.39900000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.39800000000000002,"15min":0.39300000000000002},"min":{"1min":0.28799999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.50700000000000001}]},{"osd":6,"last update":"Mon Apr 22 22:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.56799999999999995}]},{"osd":7,"last update":"Mon Apr 22 22:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.55500000000000005}]}]},{"osd":5,"up_from":37,"seq":158913789981,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5608,"kb_used_data":544,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739544,"statfs":{"total":95995035648,"available":95989293056,"internally_reserved":0,"allocated":557056,"data_stored":228786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68600000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69399999999999995,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.46300000000000002}]},{"osd":1,"last update":"Mon Apr 22 22:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67300000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.37},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.81799999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.45200000000000001}]},{"osd":2,"last update":"Mon Apr 22 22:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.38600000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.69399999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.42999999999999999}]},{"osd":3,"last update":"Mon Apr 22 22:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.73399999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.44}]},{"osd":4,"last update":"Mon Apr 22 22:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.39900000000000002,"15min":0.39700000000000002},"min":{"1min":0.30599999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.313},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41899999999999998},"min":{"1min":0.29099999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58599999999999997,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.378}]},{"osd":6,"last update":"Mon Apr 22 22:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.34799999999999998}]},{"osd":7,"last update":"Mon Apr 22 22:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.39600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T22:07:32.375 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-22T22:07:32.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:32 smithi192 ceph-mon[22835]: pgmap v42: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:33.030 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:07:33.177 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:33 smithi174 ceph-mon[21825]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:07:33.177 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:33 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:33.178 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:07:33 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:07:33] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:07:33.178 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:33 smithi174 ceph-mon[26730]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:07:33.178 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:33 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:33.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:33 smithi192 ceph-mon[22835]: from='client.24514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:07:33.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:33 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:34.133 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:34 smithi174 ceph-mon[21825]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:34.133 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:34 smithi174 ceph-mon[26730]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:34.419 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:34 smithi192 ceph-mon[22835]: pgmap v43: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:35.953 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T22:07:35.953 INFO:teuthology.orchestra.run.smithi174.stderr:dumped all 2024-04-22T22:07:36.238 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:36 smithi174 ceph-mon[21825]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:36.239 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:36 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:36.239 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:36 smithi174 ceph-mon[26730]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:36.239 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:36 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:36.239 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:07:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:07:35] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:07:36.362 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:36 smithi192 ceph-mon[22835]: pgmap v44: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:36.362 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:36 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:36.547 INFO:teuthology.orchestra.run.smithi174.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2024-04-22T22:07:35.073826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48500,"kb_used_data":5940,"kb_used_omap":0,"kb_used_meta":42496,"kb_avail":749912716,"statfs":{"total":767960285184,"available":767910621184,"internally_reserved":0,"allocated":6082560,"data_stored":3418927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43515904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002384"},"pg_stats":[{"pgid":"1.0","version":"57'128","reported_seq":266,"reported_epoch":57,"state":"active+clean","last_fresh":"2024-04-22T22:06:13.146354+0000","last_change":"2024-04-22T22:05:58.052913+0000","last_active":"2024-04-22T22:06:13.146354+0000","last_peered":"2024-04-22T22:06:13.146354+0000","last_clean":"2024-04-22T22:06:13.146354+0000","last_became_active":"2024-04-22T22:05:58.052526+0000","last_became_peered":"2024-04-22T22:05:58.052526+0000","last_unstale":"2024-04-22T22:06:13.146354+0000","last_undegraded":"2024-04-22T22:06:13.146354+0000","last_fullsized":"2024-04-22T22:06:13.146354+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-22T22:04:06.181302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-22T22:04:06.181302+0000","last_clean_scrub_stamp":"2024-04-22T22:04:06.181302+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-23T22:14:18.226886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299415,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6068,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739084,"statfs":{"total":95995035648,"available":95988822016,"internally_reserved":0,"allocated":1093632,"data_stored":757979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.48299999999999998}]},{"osd":1,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.45200000000000001}]},{"osd":2,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.435},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.47099999999999997}]},{"osd":3,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.496}]},{"osd":4,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.40000000000000002}]},{"osd":5,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.315},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.34300000000000003}]},{"osd":6,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.379}]}]},{"osd":6,"up_from":44,"seq":188978561050,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6132,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739020,"statfs":{"total":95995035648,"available":95988756480,"internally_reserved":0,"allocated":1093632,"data_stored":757979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.40500000000000003}]},{"osd":1,"last update":"Mon Apr 22 22:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.48999999999999999}]},{"osd":2,"last update":"Mon Apr 22 22:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.877,"5min":0.877,"15min":0.877},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.36799999999999999}]},{"osd":3,"last update":"Mon Apr 22 22:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.46500000000000002}]},{"osd":4,"last update":"Mon Apr 22 22:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.45400000000000001}]},{"osd":5,"last update":"Mon Apr 22 22:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.51800000000000002}]},{"osd":7,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.44},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.498}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6252,"kb_used_data":548,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738900,"statfs":{"total":95995035648,"available":95988633600,"internally_reserved":0,"allocated":561152,"data_stored":229051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.36699999999999999,"15min":0.34799999999999998},"min":{"1min":0.23499999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.5},{"interface":"front","average":{"1min":0.434,"5min":0.376,"15min":0.35199999999999998},"min":{"1min":0.28000000000000003,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.436}]},{"osd":2,"last update":"Mon Apr 22 22:07:12 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.38500000000000001,"15min":0.373},"min":{"1min":0.25700000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.439,"5min":0.40799999999999997,"15min":0.40000000000000002},"min":{"1min":0.27500000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.52000000000000002}]},{"osd":3,"last update":"Mon Apr 22 22:06:39 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.442,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.49299999999999999}]},{"osd":4,"last update":"Mon Apr 22 22:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.42899999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.61399999999999999,"15min":0.61599999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.78300000000000003,"5min":0.872,"15min":0.872},"last":0.60399999999999998}]},{"osd":5,"last update":"Mon Apr 22 22:07:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.61099999999999999,"15min":0.61899999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.59099999999999997,"15min":0.59499999999999997},"min":{"1min":0.47399999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.754,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Apr 22 22:06:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.62},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.63}]},{"osd":7,"last update":"Mon Apr 22 22:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.65700000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6900,"kb_used_data":1068,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738252,"statfs":{"total":95995035648,"available":95987970048,"internally_reserved":0,"allocated":1093632,"data_stored":757979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 22 22:06:51 2024","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.35699999999999998,"15min":0.34699999999999998},"min":{"1min":0.26300000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.47699999999999998,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36899999999999999,"15min":0.35499999999999998},"min":{"1min":0.27200000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.57799999999999996,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.34100000000000003}]},{"osd":2,"last update":"Mon Apr 22 22:07:09 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.41499999999999998,"15min":0.42199999999999999},"min":{"1min":0.246,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.48599999999999999,"5min":0.62,"15min":0.62},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.41499999999999998,"15min":0.42299999999999999},"min":{"1min":0.252,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.48499999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.312}]},{"osd":3,"last update":"Mon Apr 22 22:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.45500000000000002,"15min":0.45900000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.45000000000000001,"15min":0.45400000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.438}]},{"osd":4,"last update":"Mon Apr 22 22:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57899999999999996,"15min":0.58199999999999996},"min":{"1min":0.40000000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0229999999999999,"5min":1.0229999999999999,"15min":1.0229999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.58899999999999997,"15min":0.59699999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.81699999999999995,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.45200000000000001}]},{"osd":5,"last update":"Mon Apr 22 22:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.59799999999999998,"15min":0.59999999999999998},"min":{"1min":0.41899999999999998,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79300000000000004,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.51300000000000001}]},{"osd":6,"last update":"Mon Apr 22 22:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.53900000000000003}]},{"osd":7,"last update":"Mon Apr 22 22:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"last":0.47299999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6188,"kb_used_data":548,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":561152,"data_stored":229051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.38800000000000001,"15min":0.36299999999999999},"min":{"1min":0.28199999999999997,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.51800000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.40100000000000002,"15min":0.38700000000000001},"min":{"1min":0.29399999999999998,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.30299999999999999}]},{"osd":1,"last update":"Mon Apr 22 22:07:10 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.41099999999999998,"15min":0.39700000000000002},"min":{"1min":0.309,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.58599999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.39800000000000002,"15min":0.38100000000000001},"min":{"1min":0.29899999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"last":0.318}]},{"osd":3,"last update":"Mon Apr 22 22:06:35 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.47099999999999997,"15min":0.47199999999999998},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48399999999999999,"15min":0.48499999999999999},"min":{"1min":0.34499999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Apr 22 22:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.58299999999999996,"15min":0.57899999999999996},"min":{"1min":0.44700000000000001,"5min":0.371,"15min":0.371},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.621,"15min":0.623},"min":{"1min":0.42499999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.77400000000000002,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.49099999999999999}]},{"osd":5,"last update":"Mon Apr 22 22:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.59299999999999997,"15min":0.58999999999999997},"min":{"1min":0.40300000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.81000000000000005,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.48099999999999998}]},{"osd":6,"last update":"Mon Apr 22 22:06:35 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.55800000000000005}]},{"osd":7,"last update":"Mon Apr 22 22:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.47399999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5676,"kb_used_data":548,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739476,"statfs":{"total":95995035648,"available":95989223424,"internally_reserved":0,"allocated":561152,"data_stored":229051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.40000000000000002,"15min":0.40600000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":0.61199999999999999,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.247},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.41999999999999998,"15min":0.42099999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.60099999999999998,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.23799999999999999}]},{"osd":1,"last update":"Mon Apr 22 22:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.42899999999999999,"15min":0.42999999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58299999999999996,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.29999999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.224}]},{"osd":2,"last update":"Mon Apr 22 22:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.441,"15min":0.443},"min":{"1min":0.33900000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.61699999999999999,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.29799999999999999}]},{"osd":4,"last update":"Mon Apr 22 22:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42299999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.41099999999999998,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.379}]},{"osd":5,"last update":"Mon Apr 22 22:07:16 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.55200000000000005,"15min":0.54900000000000004},"min":{"1min":0.42799999999999999,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.73999999999999999,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.46400000000000002}]},{"osd":6,"last update":"Mon Apr 22 22:06:34 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48299999999999998}]},{"osd":7,"last update":"Mon Apr 22 22:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.49299999999999999}]}]},{"osd":4,"up_from":30,"seq":128849018915,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5676,"kb_used_data":548,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739476,"statfs":{"total":95995035648,"available":95989223424,"internally_reserved":0,"allocated":561152,"data_stored":229051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:07:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52600000000000002,"15min":0.52700000000000002},"min":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73299999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40000000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.495}]},{"osd":1,"last update":"Mon Apr 22 22:07:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.54700000000000004,"15min":0.55200000000000005},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Apr 22 22:07:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.53400000000000003,"15min":0.53500000000000003},"min":{"1min":0.39500000000000002,"5min":0.378,"15min":0.378},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.54500000000000004,"15min":0.55100000000000005},"min":{"1min":0.377,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.58599999999999997}]},{"osd":3,"last update":"Mon Apr 22 22:07:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.54000000000000004,"15min":0.54500000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.54200000000000004,"15min":0.54800000000000004},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.60599999999999998}]},{"osd":5,"last update":"Mon Apr 22 22:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.40300000000000002,"15min":0.39900000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.378},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.39800000000000002,"15min":0.39300000000000002},"min":{"1min":0.28799999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.35599999999999998}]},{"osd":6,"last update":"Mon Apr 22 22:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"last":0.40699999999999997}]},{"osd":7,"last update":"Mon Apr 22 22:06:55 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.434}]}]},{"osd":5,"up_from":37,"seq":158913789982,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5608,"kb_used_data":544,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739544,"statfs":{"total":95995035648,"available":95989293056,"internally_reserved":0,"allocated":557056,"data_stored":228786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 22 22:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.36499999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.68600000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.51800000000000002,"15min":0.52000000000000002},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69399999999999995,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.46500000000000002}]},{"osd":1,"last update":"Mon Apr 22 22:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52200000000000002,"15min":0.52900000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67300000000000004,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.5},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.81799999999999995,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Apr 22 22:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.53100000000000003,"15min":0.53500000000000003},"min":{"1min":0.38600000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.69399999999999995,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.39800000000000002}]},{"osd":3,"last update":"Mon Apr 22 22:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.55500000000000005,"15min":0.56299999999999994},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.73399999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.54900000000000004,"15min":0.55400000000000005},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.45700000000000002}]},{"osd":4,"last update":"Mon Apr 22 22:07:17 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.39900000000000002,"15min":0.39700000000000002},"min":{"1min":0.30599999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41899999999999998},"min":{"1min":0.29099999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.58599999999999997,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.33900000000000002}]},{"osd":6,"last update":"Mon Apr 22 22:06:32 2024","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.375}]},{"osd":7,"last update":"Mon Apr 22 22:06:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.36099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-22T22:07:36.549 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-22T22:07:36.549 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-22T22:07:36.549 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-22T22:07:36.549 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-22T22:07:36.964 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:36 smithi192 systemd[1]: Starting Ceph grafana.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:07:37.230 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-mon[22835]: from='client.24520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:07:37.230 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:36 smithi192 podman[41117]: 2024-04-22 22:07:36.988846293 +0000 UTC m=+0.111344404 container create 4787c8784d2e7df5eb83443bf96fde8c36e9814cca3de13ab9981cc3e6015421 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, release=236.1648460182, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.expose-services=, io.openshift.tags=base rhel8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, vendor=Red Hat, Inc., vcs-type=git, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner ) 2024-04-22T22:07:37.230 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:36 smithi192 podman[41117]: 2024-04-22 22:07:36.89459024 +0000 UTC m=+0.017088327 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-22T22:07:37.230 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 podman[41117]: 2024-04-22 22:07:37.224407003 +0000 UTC m=+0.346905072 container init 4787c8784d2e7df5eb83443bf96fde8c36e9814cca3de13ab9981cc3e6015421 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a, description=Ceph Grafana Container, vendor=Red Hat, Inc., distribution-scope=public, io.openshift.expose-services=, maintainer=Paul Cuzner , io.buildah.version=1.24.2, release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, version=8.5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, vcs-type=git, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8) 2024-04-22T22:07:37.230 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 podman[41117]: 2024-04-22 22:07:37.227059984 +0000 UTC m=+0.349558054 container start 4787c8784d2e7df5eb83443bf96fde8c36e9814cca3de13ab9981cc3e6015421 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, name=ubi8, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.buildah.version=1.24.2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=236.1648460182, io.openshift.tags=base rhel8, build-date=2022-03-28T10:36:18.413762, vcs-type=git, distribution-scope=public) 2024-04-22T22:07:37.387 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:37 smithi174 ceph-mon[21825]: from='client.24520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:07:37.388 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:37 smithi174 ceph-mon[26730]: from='client.24520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-22T22:07:37.480 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-04-22T22:07:37.480 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-04-22T22:07:37.480 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-04-22T22:07:37.480 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-04-22T22:07:37.480 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-22T22:07:37.480 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-22T22:07:37.481 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-22T22:07:37.481 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-22T22:07:37.481 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-04-22T22:07:37.481 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-04-22T22:07:37.481 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-04-22T22:07:37.481 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-04-22T22:07:37.481 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-04-22T22:07:37.481 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="App mode production" logger=settings 2024-04-22T22:07:37.481 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-04-22T22:07:37.481 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-22T22:07:37.481 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-04-22T22:07:37.481 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-04-22T22:07:37.482 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-04-22T22:07:37.483 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-04-22T22:07:37.483 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-04-22T22:07:37.483 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-04-22T22:07:37.483 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-04-22T22:07:37.483 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-04-22T22:07:37.484 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-04-22T22:07:37.485 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-04-22T22:07:37.486 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-04-22T22:07:37.487 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-04-22T22:07:37.488 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-04-22T22:07:37.489 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-04-22T22:07:37.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-04-22T22:07:37.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-04-22T22:07:37.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-04-22T22:07:37.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-04-22T22:07:37.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-04-22T22:07:37.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-04-22T22:07:37.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-04-22T22:07:37.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-04-22T22:07:37.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-04-22T22:07:37.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-22T22:07:37.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-04-22T22:07:37.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-04-22T22:07:37.490 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-04-22T22:07:37.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-04-22T22:07:37.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-04-22T22:07:37.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 bash[41117]: 4787c8784d2e7df5eb83443bf96fde8c36e9814cca3de13ab9981cc3e6015421 2024-04-22T22:07:37.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-04-22T22:07:37.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-04-22T22:07:37.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 systemd[1]: Started Ceph grafana.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:07:37.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-04-22T22:07:37.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-04-22T22:07:37.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-04-22T22:07:37.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-04-22T22:07:37.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-04-22T22:07:37.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-04-22T22:07:37.491 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-04-22T22:07:37.492 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-04-22T22:07:37.493 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-04-22T22:07:37.493 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-04-22T22:07:37.493 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-04-22T22:07:37.493 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-04-22T22:07:37.493 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-04-22T22:07:37.493 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-22T22:07:37.493 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-22T22:07:37.493 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-04-22T22:07:37.493 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-04-22T22:07:37.493 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-04-22T22:07:37.493 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-04-22T22:07:37.493 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-22T22:07:37.495 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-04-22T22:07:37.495 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-22T22:07:37.495 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-04-22T22:07:37.495 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-04-22T22:07:37.495 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-22T22:07:37.495 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-04-22T22:07:37.495 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-04-22T22:07:37.495 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-04-22T22:07:37.495 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-04-22T22:07:37.496 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-04-22T22:07:37.497 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-04-22T22:07:37.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-04-22T22:07:37.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-04-22T22:07:37.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-04-22T22:07:37.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-04-22T22:07:37.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-04-22T22:07:37.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-22T22:07:37.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-04-22T22:07:37.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-04-22T22:07:37.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-22T22:07:37.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-04-22T22:07:37.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-04-22T22:07:37.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-04-22T22:07:37.498 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-04-22T22:07:37.499 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-04-22T22:07:37.500 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-04-22T22:07:37.500 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-04-22T22:07:37.500 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-22T22:07:37.500 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-22T22:07:37.500 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-22T22:07:37.500 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-04-22T22:07:37.500 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-22T22:07:37.500 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-04-22T22:07:37.746 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-04-22T22:07:37.746 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-04-22T22:07:37.746 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-04-22T22:07:37.746 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-04-22T22:07:37.746 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-04-22T22:07:37.746 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-04-22T22:07:37.746 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-04-22T22:07:37.746 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-04-22T22:07:37.747 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-04-22T22:07:37.747 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-04-22T22:07:37.747 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-04-22T22:07:37.747 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-04-22T22:07:37.747 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-22T22:07:37.747 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-22T22:07:37.747 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-22T22:07:37.747 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-22T22:07:37.747 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-04-22T22:07:37.747 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-04-22T22:07:37.747 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-04-22T22:07:37.748 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-04-22T22:07:37.748 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-22T22:07:37.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-04-22T22:07:37.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-04-22T22:07:37.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-04-22T22:07:37.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-04-22T22:07:37.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-04-22T22:07:37.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-04-22T22:07:37.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-04-22T22:07:37.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-04-22T22:07:37.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-04-22T22:07:37.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-04-22T22:07:37.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-04-22T22:07:37.749 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-04-22T22:07:37.750 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-04-22T22:07:37.751 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-04-22T22:07:37.752 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-22T22:07:37.753 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-04-22T22:07:37.754 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-04-22T22:07:37.754 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-22T22:07:37.754 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-22T22:07:37.754 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-04-22T22:07:37.754 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-22T22:07:37.754 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-04-22T22:07:37.754 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-04-22T22:07:37.754 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-04-22T22:07:37.754 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-04-22T22:07:37.755 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-04-22T22:07:37.756 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-04-22T22:07:37.757 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-04-22T22:07:37.757 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-04-22T22:07:37.757 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-04-22T22:07:37.757 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-04-22T22:07:37.757 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-04-22T22:07:37.757 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-04-22T22:07:37.757 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-04-22T22:07:37.757 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-04-22T22:07:37.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-04-22T22:07:37.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-04-22T22:07:37.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-04-22T22:07:37.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-04-22T22:07:37.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-04-22T22:07:37.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-04-22T22:07:37.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-04-22T22:07:37.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-04-22T22:07:37.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-04-22T22:07:37.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-04-22T22:07:37.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-04-22T22:07:37.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-04-22T22:07:37.758 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-04-22T22:07:37.759 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-04-22T22:07:37.760 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-04-22T22:07:37.760 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-04-22T22:07:37.760 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-04-22T22:07:37.760 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=469.308372ms 2024-04-22T22:07:37.760 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-04-22T22:07:37.760 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-04-22T22:07:37.760 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-04-22T22:07:37.760 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-04-22T22:07:37.760 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-04-22T22:07:37.760 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-04-22T22:07:38.029 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-04-22T22:07:38.029 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-22T22:07:38.029 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-04-22T22:07:38.029 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-04-22T22:07:38.030 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:07:37 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:07:37+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-04-22T22:07:38.165 INFO:teuthology.orchestra.run.smithi174.stderr:Inferring config /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/mon.a/config 2024-04-22T22:07:38.288 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:38 smithi192 ceph-mon[22835]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:38.289 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:38 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.289 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:38 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:07:38.289 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:38 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:07:38.289 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:38 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:07:38.289 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:38 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.289 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:38 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.289 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:38 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.289 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:38 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.290 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[26730]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:38.290 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.290 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:07:38.290 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:07:38.290 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:07:38.290 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.290 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.290 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.290 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.290 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[21825]: pgmap v45: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:38.291 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.291 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-22T22:07:38.291 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:07:38.291 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:07:38.291 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.291 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.291 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.291 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:38 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:38.291 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:07:37 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:07:37] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:07:39.184 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:39 smithi174 ceph-mon[21825]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:07:39.184 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:39 smithi174 ceph-mon[21825]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-22T22:07:39.184 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:39 smithi174 ceph-mon[21825]: Reconfiguring daemon alertmanager.a on smithi174 2024-04-22T22:07:39.185 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:39 smithi174 ceph-mon[26730]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:07:39.185 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:39 smithi174 ceph-mon[26730]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-22T22:07:39.185 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:39 smithi174 ceph-mon[26730]: Reconfiguring daemon alertmanager.a on smithi174 2024-04-22T22:07:39.292 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:39 smithi192 ceph-mon[22835]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:07:39.293 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:39 smithi192 ceph-mon[22835]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-22T22:07:39.293 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:39 smithi192 ceph-mon[22835]: Reconfiguring daemon alertmanager.a on smithi174 2024-04-22T22:07:40.225 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:40 smithi174 ceph-mon[21825]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:40.226 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:40 smithi174 ceph-mon[26730]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:40.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:40 smithi192 ceph-mon[22835]: pgmap v46: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:41.449 INFO:teuthology.orchestra.run.smithi174.stdout: 2024-04-22T22:07:41.449 INFO:teuthology.orchestra.run.smithi174.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-22T22:07:41.449 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:07:41 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:07:41] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:07:42.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:42 smithi174 ceph-mon[21825]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:42.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:42 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:42.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:42 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1266995785' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T22:07:42.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:42 smithi192 ceph-mon[22835]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:42.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:42 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:42.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:42 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1266995785' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T22:07:42.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:42 smithi174 ceph-mon[26730]: pgmap v47: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:42.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:42 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:42.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:42 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1266995785' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-22T22:07:43.380 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-22T22:07:43.380 INFO:tasks.cephadm:Setup complete, yielding 2024-04-22T22:07:43.381 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T22:07:43.390 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi174.front.sepia.ceph.com 2024-04-22T22:07:43.390 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest false --force' 2024-04-22T22:07:43.917 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:07:43 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:07:43] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:07:44.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:44 smithi174 ceph-mon[26730]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:44.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:44.237 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:44.237 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:44 smithi174 ceph-mon[21825]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:44.237 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:44.237 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:44.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:44 smithi192 ceph-mon[22835]: pgmap v48: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:44.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:44.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:45.064 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:45 smithi174 systemd[1]: Stopping Ceph alertmanager.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:07:46.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:46 smithi174 ceph-mon[21825]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:46.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:46 smithi174 ceph-mon[26730]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:46.316 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:46 smithi174 bash[47316]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-22T22:07:46.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:46 smithi192 ceph-mon[22835]: pgmap v49: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:46.736 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:07:46 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:07:46] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:07:47.065 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:46 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[45427]: level=info ts=2024-04-22T22:07:46.761Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-22T22:07:47.065 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:46 smithi174 podman[47359]: 2024-04-22 22:07:46.791904541 +0000 UTC m=+0.608682615 container died ca09846bf6a44718bc5bb12b4175738541a9ee1718fe483eb144f2000b63eb85 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:47.496 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:47 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:47.497 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:47 smithi174 ceph-mon[21825]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:47.497 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:47 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:47.497 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:47 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3260411540' entity='client.admin' 2024-04-22T22:07:47.497 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:47 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:47.497 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:47 smithi174 ceph-mon[26730]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:47.497 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:47 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:47.497 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:47 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3260411540' entity='client.admin' 2024-04-22T22:07:47.498 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:47 smithi174 podman[47359]: 2024-04-22 22:07:47.278461602 +0000 UTC m=+1.095239718 container cleanup ca09846bf6a44718bc5bb12b4175738541a9ee1718fe483eb144f2000b63eb85 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:47.498 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:47 smithi174 bash[47359]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a 2024-04-22T22:07:47.498 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:47 smithi174 bash[47484]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-22T22:07:47.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:47 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:47.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:47 smithi192 ceph-mon[22835]: pgmap v50: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:47.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:47 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:47.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:47 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3260411540' entity='client.admin' 2024-04-22T22:07:47.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:47 smithi174 podman[47455]: 2024-04-22 22:07:47.521519498 +0000 UTC m=+0.727700288 container remove ca09846bf6a44718bc5bb12b4175738541a9ee1718fe483eb144f2000b63eb85 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:47.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:47 smithi174 podman[47455]: 2024-04-22 22:07:47.630773373 +0000 UTC m=+0.836954124 volume remove 6062fcd25f81c78a9b9e5fc23fd66838383cbe8934da33dc42ea45357894f16f 2024-04-22T22:07:47.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:47 smithi174 systemd[1]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-22T22:07:47.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:47 smithi174 systemd[1]: Stopped Ceph alertmanager.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:07:47.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:47 smithi174 systemd[1]: Starting Ceph alertmanager.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:07:48.302 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T22:07:48.312 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi174.front.sepia.ceph.com 2024-04-22T22:07:48.312 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-04-22T22:07:48.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:48 smithi174 ceph-mon[21825]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-22T22:07:48.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:48 smithi174 ceph-mon[26730]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-22T22:07:48.565 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:48 smithi174 podman[47570]: 2024-04-22 22:07:48.380809827 +0000 UTC m=+0.019353596 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-22T22:07:48.565 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:48 smithi174 podman[47570]: 2024-04-22 22:07:48.493418112 +0000 UTC m=+0.131961873 volume create 93fa0454b17ff2f97634cfaa9f8620f0fb619ea685ad5623b23a6c7eb8a0eff2 2024-04-22T22:07:48.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:48 smithi192 ceph-mon[22835]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-22T22:07:48.896 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:48 smithi174 podman[47570]: 2024-04-22 22:07:48.593948439 +0000 UTC m=+0.232492200 container create 1e9de7a331d8e2f1ed2c169d855595d5245cdb6d3298a8b42c3f89debd3b8bb6 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:48.896 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:48 smithi174 podman[47570]: 2024-04-22 22:07:48.921529385 +0000 UTC m=+0.560073149 container init 1e9de7a331d8e2f1ed2c169d855595d5245cdb6d3298a8b42c3f89debd3b8bb6 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:49.160 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:48 smithi174 podman[47570]: 2024-04-22 22:07:48.924222369 +0000 UTC m=+0.562766131 container start 1e9de7a331d8e2f1ed2c169d855595d5245cdb6d3298a8b42c3f89debd3b8bb6 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:49.160 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:48 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=info ts=2024-04-22T22:07:48.934Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-22T22:07:49.160 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:48 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=info ts=2024-04-22T22:07:48.934Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-22T22:07:49.160 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:48 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=info ts=2024-04-22T22:07:48.935Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.174 port=9094 2024-04-22T22:07:49.160 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:48 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=info ts=2024-04-22T22:07:48.937Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-22T22:07:49.160 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:48 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=info ts=2024-04-22T22:07:48.961Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-22T22:07:49.160 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:48 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=info ts=2024-04-22T22:07:48.961Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-22T22:07:49.160 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:48 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=info ts=2024-04-22T22:07:48.964Z caller=main.go:518 msg=Listening address=:9093 2024-04-22T22:07:49.160 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:48 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=info ts=2024-04-22T22:07:48.964Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-22T22:07:49.160 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:49 smithi174 bash[47570]: 1e9de7a331d8e2f1ed2c169d855595d5245cdb6d3298a8b42c3f89debd3b8bb6 2024-04-22T22:07:49.160 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:49 smithi174 systemd[1]: Started Ceph alertmanager.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:07:50.154 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:50 smithi174 ceph-mon[21825]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:50.154 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:50 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:50.154 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:50 smithi174 ceph-mon[21825]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-22T22:07:50.154 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:50 smithi174 ceph-mon[21825]: Reconfiguring daemon prometheus.a on smithi192 2024-04-22T22:07:50.155 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:50 smithi174 ceph-mon[26730]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:50.155 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:50 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:50.155 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:50 smithi174 ceph-mon[26730]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-22T22:07:50.155 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:50 smithi174 ceph-mon[26730]: Reconfiguring daemon prometheus.a on smithi192 2024-04-22T22:07:50.290 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:50 smithi192 ceph-mon[22835]: pgmap v51: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:50.291 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:50 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:50.291 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:50 smithi192 ceph-mon[22835]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-22T22:07:50.291 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:50 smithi192 ceph-mon[22835]: Reconfiguring daemon prometheus.a on smithi192 2024-04-22T22:07:51.315 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:50 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=info ts=2024-04-22T22:07:50.937Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000095576s 2024-04-22T22:07:52.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:52 smithi174 ceph-mon[21825]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:52.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:52 smithi174 ceph-mon[26730]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:52.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:52 smithi192 ceph-mon[22835]: pgmap v52: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:53.068 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-22T22:07:53.068 INFO:teuthology.orchestra.run.smithi174.stdout: "id": "aee65c58-0249-4dc9-af4f-d5a49cb96746", 2024-04-22T22:07:53.068 INFO:teuthology.orchestra.run.smithi174.stdout: "name": "r", 2024-04-22T22:07:53.068 INFO:teuthology.orchestra.run.smithi174.stdout: "current_period": "d6d11f22-fe51-4cbf-b58c-ead12aff539f", 2024-04-22T22:07:53.068 INFO:teuthology.orchestra.run.smithi174.stdout: "epoch": 1 2024-04-22T22:07:53.068 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-22T22:07:53.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:53 smithi192 ceph-mon[22835]: osdmap e58: 8 total, 8 up, 8 in 2024-04-22T22:07:53.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:53 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3343728952' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-22T22:07:53.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:53 smithi192 ceph-mon[22835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-22T22:07:53.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:53 smithi192 ceph-mon[22835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-22T22:07:53.255 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:53 smithi192 ceph-mon[22835]: osdmap e59: 8 total, 8 up, 8 in 2024-04-22T22:07:53.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:53 smithi174 ceph-mon[26730]: osdmap e58: 8 total, 8 up, 8 in 2024-04-22T22:07:53.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:53 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3343728952' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-22T22:07:53.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:53 smithi174 ceph-mon[26730]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-22T22:07:53.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:53 smithi174 ceph-mon[26730]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-22T22:07:53.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:53 smithi174 ceph-mon[26730]: osdmap e59: 8 total, 8 up, 8 in 2024-04-22T22:07:53.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:53 smithi174 ceph-mon[21825]: osdmap e58: 8 total, 8 up, 8 in 2024-04-22T22:07:53.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:53 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3343728952' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-22T22:07:53.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:53 smithi174 ceph-mon[21825]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-22T22:07:53.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:53 smithi174 ceph-mon[21825]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-22T22:07:53.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:53 smithi174 ceph-mon[21825]: osdmap e59: 8 total, 8 up, 8 in 2024-04-22T22:07:53.782 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-04-22T22:07:54.282 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-mon[22835]: pgmap v54: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:54.282 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-mon[22835]: osdmap e60: 8 total, 8 up, 8 in 2024-04-22T22:07:54.283 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 systemd[1]: Stopping Ceph prometheus.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:07:54.283 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 bash[41950]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-22T22:07:54.283 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:07:54.180Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-22T22:07:54.283 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:07:54.180Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-22T22:07:54.283 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:07:54.180Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-22T22:07:54.283 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:07:54.180Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-22T22:07:54.283 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:07:54.180Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-22T22:07:54.283 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:07:54.180Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-22T22:07:54.283 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:07:54.180Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-22T22:07:54.283 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:07:54.180Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-22T22:07:54.283 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:07:54.180Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-22T22:07:54.283 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:07:54.181Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-22T22:07:54.283 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:07:54.181Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-22T22:07:54.284 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[38835]: ts=2024-04-22T22:07:54.181Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-22T22:07:54.284 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 podman[41962]: 2024-04-22 22:07:54.190593063 +0000 UTC m=+0.137629214 container died 66f63bfb31b95157691f121fc8a29d2d3150887d63b049b494217fe958005abf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:54.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:54 smithi174 ceph-mon[21825]: pgmap v54: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:54.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:54 smithi174 ceph-mon[21825]: osdmap e60: 8 total, 8 up, 8 in 2024-04-22T22:07:54.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:54 smithi174 ceph-mon[26730]: pgmap v54: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:07:54.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:54 smithi174 ceph-mon[26730]: osdmap e60: 8 total, 8 up, 8 in 2024-04-22T22:07:54.806 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 podman[41962]: 2024-04-22 22:07:54.563175698 +0000 UTC m=+0.510211854 container cleanup 66f63bfb31b95157691f121fc8a29d2d3150887d63b049b494217fe958005abf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:54.806 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 bash[41962]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a 2024-04-22T22:07:54.806 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 bash[41990]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-22T22:07:55.057 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 podman[41975]: 2024-04-22 22:07:54.831219837 +0000 UTC m=+0.642866099 container remove 66f63bfb31b95157691f121fc8a29d2d3150887d63b049b494217fe958005abf (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:55.057 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 systemd[1]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-22T22:07:55.057 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 systemd[1]: Stopped Ceph prometheus.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:07:55.057 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:54 smithi192 systemd[1]: Starting Ceph prometheus.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:07:55.057 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 podman[42066]: 2024-04-22 22:07:54.960377692 +0000 UTC m=+0.019563269 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-22T22:07:55.057 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 podman[42066]: 2024-04-22 22:07:55.082264513 +0000 UTC m=+0.141450089 container create dfa924a46ee2d26e0516c8747b982f729089d1559bcc8fde454576be9f638793 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:55.309 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 podman[42066]: 2024-04-22 22:07:55.275979344 +0000 UTC m=+0.335164920 container init dfa924a46ee2d26e0516c8747b982f729089d1559bcc8fde454576be9f638793 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:55.309 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 podman[42066]: 2024-04-22 22:07:55.278636579 +0000 UTC m=+0.337822164 container start dfa924a46ee2d26e0516c8747b982f729089d1559bcc8fde454576be9f638793 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T22:07:55.309 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.310Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-22T22:07:55.309 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.310Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-22T22:07:55.309 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.310Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-22T22:07:55.309 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.310Z caller=main.go:518 level=info host_details="(Linux 5.14.0-437.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Apr 9 12:57:02 UTC 2024 x86_64 smithi192 (none))" 2024-04-22T22:07:55.309 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.310Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-22T22:07:55.309 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.310Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-22T22:07:55.309 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.311Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-22T22:07:55.310 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.312Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-22T22:07:55.310 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.313Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-22T22:07:55.310 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.315Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-22T22:07:55.310 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.315Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.741µs 2024-04-22T22:07:55.310 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.315Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-22T22:07:55.310 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.317Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-22T22:07:55.310 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.317Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-22T22:07:55.310 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.317Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=26.823µs wal_replay_duration=1.582558ms total_replay_duration=1.624481ms 2024-04-22T22:07:55.310 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.318Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-22T22:07:55.310 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.318Z caller=main.go:947 level=info msg="TSDB started" 2024-04-22T22:07:55.310 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.318Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-22T22:07:55.310 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.334Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.821488ms db_storage=730ns remote_storage=1.627µs web_handler=311ns query_engine=647ns scrape=327.855µs scrape_sd=21.784µs notify=24.239µs notify_sd=7.39µs rules=15.221433ms 2024-04-22T22:07:55.310 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:07:55.334Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-22T22:07:55.564 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 bash[42066]: dfa924a46ee2d26e0516c8747b982f729089d1559bcc8fde454576be9f638793 2024-04-22T22:07:55.564 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:07:55 smithi192 systemd[1]: Started Ceph prometheus.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:07:55.603 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-22T22:07:55.603 INFO:teuthology.orchestra.run.smithi174.stdout: "id": "d4ed8f16-6160-46c2-8438-40b725a3d6dc", 2024-04-22T22:07:55.603 INFO:teuthology.orchestra.run.smithi174.stdout: "name": "default", 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout: "api_name": "default", 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout: "is_master": "true", 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout: "endpoints": [], 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout: "hostnames": [], 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout: "hostnames_s3website": [], 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout: "master_zone": "", 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout: "zones": [], 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout: "placement_targets": [], 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout: "default_placement": "", 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout: "realm_id": "aee65c58-0249-4dc9-af4f-d5a49cb96746", 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout: "sync_policy": { 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout: "groups": [] 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-22T22:07:55.604 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: pgmap v57: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.174:9093"}]: dispatch 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.192:3000"}]: dispatch 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:07:56.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:56 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: pgmap v57: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-22T22:07:56.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T22:07:56.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.174:9093"}]: dispatch 2024-04-22T22:07:56.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:07:56.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T22:07:56.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.192:3000"}]: dispatch 2024-04-22T22:07:56.316 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:07:56.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:07:56.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:07:56.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:07:56.317 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: pgmap v57: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 1 op/s 2024-04-22T22:07:56.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T22:07:56.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.174:9093"}]: dispatch 2024-04-22T22:07:56.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:07:56.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T22:07:56.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.192:3000"}]: dispatch 2024-04-22T22:07:56.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:07:56.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:07:56.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:07:56.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:07:56.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:56 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:07:56.369 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-04-22T22:07:57.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[21825]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T22:07:57.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[21825]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.174:9093"}]: dispatch 2024-04-22T22:07:57.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[21825]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:07:57.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[21825]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T22:07:57.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[21825]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.192:3000"}]: dispatch 2024-04-22T22:07:57.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[21825]: pgmap v58: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 2 op/s 2024-04-22T22:07:57.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[21825]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-22T22:07:57.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[21825]: Cluster is now healthy 2024-04-22T22:07:57.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[26730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T22:07:57.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[26730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.174:9093"}]: dispatch 2024-04-22T22:07:57.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[26730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:07:57.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[26730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T22:07:57.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[26730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.192:3000"}]: dispatch 2024-04-22T22:07:57.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[26730]: pgmap v58: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 2 op/s 2024-04-22T22:07:57.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[26730]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-22T22:07:57.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:57 smithi174 ceph-mon[26730]: Cluster is now healthy 2024-04-22T22:07:57.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:57 smithi192 ceph-mon[22835]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-22T22:07:57.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:57 smithi192 ceph-mon[22835]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.174:9093"}]: dispatch 2024-04-22T22:07:57.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:57 smithi192 ceph-mon[22835]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-22T22:07:57.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:57 smithi192 ceph-mon[22835]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-22T22:07:57.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:57 smithi192 ceph-mon[22835]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.192:3000"}]: dispatch 2024-04-22T22:07:57.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:57 smithi192 ceph-mon[22835]: pgmap v58: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 2 op/s 2024-04-22T22:07:57.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:57 smithi192 ceph-mon[22835]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-22T22:07:57.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:57 smithi192 ceph-mon[22835]: Cluster is now healthy 2024-04-22T22:07:58.039 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-22T22:07:58.039 INFO:teuthology.orchestra.run.smithi174.stdout: "id": "098ea965-b2fc-4978-a8a5-158355c01da6", 2024-04-22T22:07:58.039 INFO:teuthology.orchestra.run.smithi174.stdout: "name": "z", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "domain_root": "z.rgw.meta:root", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "control_pool": "z.rgw.control", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "gc_pool": "z.rgw.log:gc", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "lc_pool": "z.rgw.log:lc", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "log_pool": "z.rgw.log", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "roles_pool": "z.rgw.meta:roles", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "otp_pool": "z.rgw.otp", 2024-04-22T22:07:58.040 INFO:teuthology.orchestra.run.smithi174.stdout: "system_key": { 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: "access_key": "", 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: "secret_key": "" 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: "placement_pools": [ 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: "key": "default-placement", 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: "val": { 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: "index_pool": "z.rgw.buckets.index", 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: "storage_classes": { 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: "STANDARD": { 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: "data_pool": "z.rgw.buckets.data" 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: "index_type": 0 2024-04-22T22:07:58.041 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-22T22:07:58.042 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-22T22:07:58.042 INFO:teuthology.orchestra.run.smithi174.stdout: ], 2024-04-22T22:07:58.042 INFO:teuthology.orchestra.run.smithi174.stdout: "realm_id": "aee65c58-0249-4dc9-af4f-d5a49cb96746", 2024-04-22T22:07:58.042 INFO:teuthology.orchestra.run.smithi174.stdout: "notif_pool": "z.rgw.log:notif" 2024-04-22T22:07:58.042 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-22T22:07:58.784 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-04-22T22:07:59.057 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:07:58 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=info ts=2024-04-22T22:07:58.939Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002233872s 2024-04-22T22:07:59.312 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:07:59 smithi192 ceph-mon[22835]: pgmap v59: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 2 op/s 2024-04-22T22:07:59.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:07:59 smithi174 ceph-mon[21825]: pgmap v59: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 2 op/s 2024-04-22T22:07:59.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:07:59 smithi174 ceph-mon[26730]: pgmap v59: 33 pgs: 33 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 2 op/s 2024-04-22T22:08:01.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:01 smithi192 ceph-mon[22835]: pgmap v60: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1020 B/s rd, 2.3 KiB/s wr, 3 op/s 2024-04-22T22:08:01.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:01 smithi174 ceph-mon[21825]: pgmap v60: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1020 B/s rd, 2.3 KiB/s wr, 3 op/s 2024-04-22T22:08:01.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:01 smithi174 ceph-mon[26730]: pgmap v60: 33 pgs: 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1020 B/s rd, 2.3 KiB/s wr, 3 op/s 2024-04-22T22:08:02.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:02 smithi192 ceph-mon[22835]: osdmap e61: 8 total, 8 up, 8 in 2024-04-22T22:08:02.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:02 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1327154355' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-22T22:08:02.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:02 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:08:02] "GET /metrics HTTP/1.1" 200 192870 "" "Prometheus/2.33.4" 2024-04-22T22:08:02.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:02 smithi174 ceph-mon[26730]: osdmap e61: 8 total, 8 up, 8 in 2024-04-22T22:08:02.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:02 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1327154355' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-22T22:08:02.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:02 smithi174 ceph-mon[21825]: osdmap e61: 8 total, 8 up, 8 in 2024-04-22T22:08:02.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:02 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1327154355' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-22T22:08:02.814 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:08:02 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:08:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:08:03.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:03 smithi174 ceph-mon[26730]: pgmap v62: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-22T22:08:03.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:03 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1327154355' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-22T22:08:03.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:03 smithi174 ceph-mon[26730]: osdmap e62: 8 total, 8 up, 8 in 2024-04-22T22:08:03.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:03 smithi174 ceph-mon[21825]: pgmap v62: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-22T22:08:03.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:03 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1327154355' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-22T22:08:03.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:03 smithi174 ceph-mon[21825]: osdmap e62: 8 total, 8 up, 8 in 2024-04-22T22:08:03.380 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:03 smithi192 ceph-mon[22835]: pgmap v62: 65 pgs: 32 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 976 B/s rd, 2.2 KiB/s wr, 3 op/s 2024-04-22T22:08:03.380 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:03 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1327154355' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-22T22:08:03.380 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:03 smithi192 ceph-mon[22835]: osdmap e62: 8 total, 8 up, 8 in 2024-04-22T22:08:03.814 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:03 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:08:03] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:04.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:04 smithi192 ceph-mon[22835]: osdmap e63: 8 total, 8 up, 8 in 2024-04-22T22:08:04.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:04 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1327154355' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-22T22:08:04.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:04 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:04.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:04 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:04.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:04 smithi174 ceph-mon[26730]: osdmap e63: 8 total, 8 up, 8 in 2024-04-22T22:08:04.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:04 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1327154355' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-22T22:08:04.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:04 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:04.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:04 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:04.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:04 smithi174 ceph-mon[21825]: osdmap e63: 8 total, 8 up, 8 in 2024-04-22T22:08:04.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:04 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1327154355' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-22T22:08:04.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:04 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:04.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:04 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:05.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:05 smithi192 ceph-mon[22835]: pgmap v65: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-04-22T22:08:05.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:05 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1327154355' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-22T22:08:05.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:05 smithi192 ceph-mon[22835]: osdmap e64: 8 total, 8 up, 8 in 2024-04-22T22:08:05.381 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:05 smithi174 ceph-mon[21825]: pgmap v65: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-04-22T22:08:05.381 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:05 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1327154355' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-22T22:08:05.381 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:05 smithi174 ceph-mon[21825]: osdmap e64: 8 total, 8 up, 8 in 2024-04-22T22:08:05.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:05 smithi174 ceph-mon[26730]: pgmap v65: 97 pgs: 64 unknown, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-04-22T22:08:05.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:05 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1327154355' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-22T22:08:05.382 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:05 smithi174 ceph-mon[26730]: osdmap e64: 8 total, 8 up, 8 in 2024-04-22T22:08:05.382 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:05 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:08:05] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:06.481 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:06 smithi174 ceph-mon[21825]: osdmap e65: 8 total, 8 up, 8 in 2024-04-22T22:08:06.481 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:06 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-22T22:08:06.481 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:06 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:06.481 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:06 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:06.481 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:06 smithi174 ceph-mon[26730]: osdmap e65: 8 total, 8 up, 8 in 2024-04-22T22:08:06.481 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:06 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-22T22:08:06.481 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:06 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:06.481 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:06 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:06.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:06 smithi192 ceph-mon[22835]: osdmap e65: 8 total, 8 up, 8 in 2024-04-22T22:08:06.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:06 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-22T22:08:06.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:06 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:06.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:06 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:06.758 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:06 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:08:06] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:07.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[21825]: pgmap v68: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-04-22T22:08:07.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-22T22:08:07.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[21825]: osdmap e66: 8 total, 8 up, 8 in 2024-04-22T22:08:07.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-22T22:08:07.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:07.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:07.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:07.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:07.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:07.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[26730]: pgmap v68: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-04-22T22:08:07.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-22T22:08:07.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[26730]: osdmap e66: 8 total, 8 up, 8 in 2024-04-22T22:08:07.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-22T22:08:07.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:07.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:07.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:07.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:07.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:07 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:07.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:07 smithi192 ceph-mon[22835]: pgmap v68: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 5.7 KiB/s rd, 5 op/s 2024-04-22T22:08:07.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:07 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-22T22:08:07.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:07 smithi192 ceph-mon[22835]: osdmap e66: 8 total, 8 up, 8 in 2024-04-22T22:08:07.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:07 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-22T22:08:07.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:07 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:07.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:07 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:07.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:07 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:07.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:07 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:07.567 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:07 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:08.176 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-22T22:08:08.176 INFO:teuthology.orchestra.run.smithi174.stdout: "id": "d2e4fefb-9352-4865-a9e8-da289f0597b5", 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "epoch": 1, 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "predecessor_uuid": "d6d11f22-fe51-4cbf-b58c-ead12aff539f", 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "sync_status": [], 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "period_map": { 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "id": "d2e4fefb-9352-4865-a9e8-da289f0597b5", 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "zonegroups": [ 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "id": "d4ed8f16-6160-46c2-8438-40b725a3d6dc", 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "name": "default", 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "api_name": "default", 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "is_master": "true", 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "endpoints": [], 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "hostnames": [], 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "hostnames_s3website": [], 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "master_zone": "098ea965-b2fc-4978-a8a5-158355c01da6", 2024-04-22T22:08:08.177 INFO:teuthology.orchestra.run.smithi174.stdout: "zones": [ 2024-04-22T22:08:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-04-22T22:08:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "id": "098ea965-b2fc-4978-a8a5-158355c01da6", 2024-04-22T22:08:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "name": "z", 2024-04-22T22:08:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "endpoints": [], 2024-04-22T22:08:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "log_meta": "false", 2024-04-22T22:08:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "log_data": "false", 2024-04-22T22:08:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "bucket_index_max_shards": 11, 2024-04-22T22:08:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "read_only": "false", 2024-04-22T22:08:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "tier_type": "", 2024-04-22T22:08:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "sync_from_all": "true", 2024-04-22T22:08:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "sync_from": [], 2024-04-22T22:08:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: "redirect_zone": "" 2024-04-22T22:08:08.178 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: ], 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "placement_targets": [ 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "name": "default-placement", 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "tags": [], 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "storage_classes": [ 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "STANDARD" 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: ], 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "default_placement": "default-placement", 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "realm_id": "aee65c58-0249-4dc9-af4f-d5a49cb96746", 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "sync_policy": { 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: "groups": [] 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-22T22:08:08.179 INFO:teuthology.orchestra.run.smithi174.stdout: ], 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "short_zone_ids": [ 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: { 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "key": "098ea965-b2fc-4978-a8a5-158355c01da6", 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "val": 2876304401 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: ] 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "master_zonegroup": "d4ed8f16-6160-46c2-8438-40b725a3d6dc", 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "master_zone": "098ea965-b2fc-4978-a8a5-158355c01da6", 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "period_config": { 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "bucket_quota": { 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "enabled": false, 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "check_on_raw": false, 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "max_size": -1, 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "max_size_kb": 0, 2024-04-22T22:08:08.180 INFO:teuthology.orchestra.run.smithi174.stdout: "max_objects": -1 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "user_quota": { 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "enabled": false, 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "check_on_raw": false, 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "max_size": -1, 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "max_size_kb": 0, 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "max_objects": -1 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "user_ratelimit": { 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "max_read_ops": 0, 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "max_write_ops": 0, 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "max_read_bytes": 0, 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "max_write_bytes": 0, 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "enabled": false 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "bucket_ratelimit": { 2024-04-22T22:08:08.181 INFO:teuthology.orchestra.run.smithi174.stdout: "max_read_ops": 0, 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "max_write_ops": 0, 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "max_read_bytes": 0, 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "max_write_bytes": 0, 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "enabled": false 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "anonymous_ratelimit": { 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "max_read_ops": 0, 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "max_write_ops": 0, 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "max_read_bytes": 0, 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "max_write_bytes": 0, 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "enabled": false 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "realm_id": "aee65c58-0249-4dc9-af4f-d5a49cb96746", 2024-04-22T22:08:08.182 INFO:teuthology.orchestra.run.smithi174.stdout: "realm_name": "r", 2024-04-22T22:08:08.183 INFO:teuthology.orchestra.run.smithi174.stdout: "realm_epoch": 2 2024-04-22T22:08:08.183 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-22T22:08:08.287 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:08 smithi174 ceph-mon[21825]: pgmap v70: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 6 op/s 2024-04-22T22:08:08.287 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:08 smithi174 ceph-mon[21825]: pgmap v71: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 7 op/s 2024-04-22T22:08:08.287 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:08 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-22T22:08:08.287 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:08 smithi174 ceph-mon[21825]: osdmap e67: 8 total, 8 up, 8 in 2024-04-22T22:08:08.288 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:08 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-22T22:08:08.288 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:08 smithi174 ceph-mon[21825]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-22T22:08:08.288 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:08 smithi174 ceph-mon[26730]: pgmap v70: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 6 op/s 2024-04-22T22:08:08.288 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:08 smithi174 ceph-mon[26730]: pgmap v71: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 7 op/s 2024-04-22T22:08:08.288 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:08 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-22T22:08:08.288 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:08 smithi174 ceph-mon[26730]: osdmap e67: 8 total, 8 up, 8 in 2024-04-22T22:08:08.288 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:08 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-22T22:08:08.288 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:08 smithi174 ceph-mon[26730]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-22T22:08:08.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:08 smithi192 ceph-mon[22835]: pgmap v70: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 6.8 KiB/s rd, 6 op/s 2024-04-22T22:08:08.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:08 smithi192 ceph-mon[22835]: pgmap v71: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 7.4 KiB/s rd, 7 op/s 2024-04-22T22:08:08.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:08 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-22T22:08:08.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:08 smithi192 ceph-mon[22835]: osdmap e67: 8 total, 8 up, 8 in 2024-04-22T22:08:08.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:08 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-22T22:08:08.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:08 smithi192 ceph-mon[22835]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-04-22T22:08:08.995 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-04-22T22:08:09.252 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:09 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-22T22:08:09.252 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:09 smithi174 ceph-mon[21825]: osdmap e68: 8 total, 8 up, 8 in 2024-04-22T22:08:09.252 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:09 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-22T22:08:09.252 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:09 smithi174 ceph-mon[26730]: osdmap e68: 8 total, 8 up, 8 in 2024-04-22T22:08:09.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:09 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2982482098' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-22T22:08:09.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:09 smithi192 ceph-mon[22835]: osdmap e68: 8 total, 8 up, 8 in 2024-04-22T22:08:10.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:10 smithi174 ceph-mon[21825]: pgmap v74: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:10.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:10 smithi174 ceph-mon[26730]: pgmap v74: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:10.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:10 smithi192 ceph-mon[22835]: pgmap v74: 129 pgs: 32 unknown, 32 creating+peering, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:12.173 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled rgw.foo update... 2024-04-22T22:08:12.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:12 smithi192 ceph-mon[22835]: pgmap v75: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-04-22T22:08:12.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:12 smithi174 ceph-mon[21825]: pgmap v75: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-04-22T22:08:12.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:12 smithi174 ceph-mon[26730]: pgmap v75: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-04-22T22:08:12.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:12 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:08:12] "GET /metrics HTTP/1.1" 200 192870 "" "Prometheus/2.33.4" 2024-04-22T22:08:12.814 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:08:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:08:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:08:12.860 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:12 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:08:12] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:13.362 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch apply rgw smpl' 2024-04-22T22:08:13.383 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:08:13.383 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: Saving service rgw.foo spec with placement count:2 2024-04-22T22:08:13.383 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.383 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:13.383 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:13.383 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.383 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.383 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi174.eqkeoc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi174.eqkeoc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:08:13.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: Saving service rgw.foo spec with placement count:2 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi174.eqkeoc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi174.eqkeoc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:13.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-04-22T22:08:13.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-04-22T22:08:13.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-22T22:08:13.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-22T22:08:13.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:08:13.386 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:08:13.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='client.24595 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: Saving service rgw.foo spec with placement count:2 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi174.eqkeoc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi174.eqkeoc", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-04-22T22:08:13.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-04-22T22:08:13.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-04-22T22:08:13.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-04-22T22:08:13.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:13.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:08:13.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:08:14.421 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[26730]: pgmap v76: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-22T22:08:14.422 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[26730]: pgmap v77: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-22T22:08:14.422 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[26730]: Saving service rgw.foo spec with placement count:2 2024-04-22T22:08:14.422 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[26730]: Deploying daemon rgw.foo.smithi174.eqkeoc on smithi174 2024-04-22T22:08:14.422 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[26730]: pgmap v78: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-22T22:08:14.422 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[26730]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-22T22:08:14.422 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[26730]: Cluster is now healthy 2024-04-22T22:08:14.422 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-04-22T22:08:14.422 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-04-22T22:08:14.422 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-22T22:08:14.422 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-04-22T22:08:14.422 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[26730]: osdmap e69: 8 total, 8 up, 8 in 2024-04-22T22:08:14.422 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[21825]: pgmap v76: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-22T22:08:14.423 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[21825]: pgmap v77: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-22T22:08:14.423 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[21825]: Saving service rgw.foo spec with placement count:2 2024-04-22T22:08:14.423 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[21825]: Deploying daemon rgw.foo.smithi174.eqkeoc on smithi174 2024-04-22T22:08:14.423 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[21825]: pgmap v78: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-22T22:08:14.423 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[21825]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-22T22:08:14.423 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[21825]: Cluster is now healthy 2024-04-22T22:08:14.423 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-04-22T22:08:14.423 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-04-22T22:08:14.423 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-22T22:08:14.423 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-04-22T22:08:14.423 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:14 smithi174 ceph-mon[21825]: osdmap e69: 8 total, 8 up, 8 in 2024-04-22T22:08:14.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:14 smithi192 ceph-mon[22835]: pgmap v76: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-22T22:08:14.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:14 smithi192 ceph-mon[22835]: pgmap v77: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-22T22:08:14.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:14 smithi192 ceph-mon[22835]: Saving service rgw.foo spec with placement count:2 2024-04-22T22:08:14.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:14 smithi192 ceph-mon[22835]: Deploying daemon rgw.foo.smithi174.eqkeoc on smithi174 2024-04-22T22:08:14.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:14 smithi192 ceph-mon[22835]: pgmap v78: 129 pgs: 129 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.6 KiB/s wr, 4 op/s 2024-04-22T22:08:14.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:14 smithi192 ceph-mon[22835]: Health check cleared: CEPHADM_FAILED_DAEMON (was: 1 failed cephadm daemon(s)) 2024-04-22T22:08:14.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:14 smithi192 ceph-mon[22835]: Cluster is now healthy 2024-04-22T22:08:14.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-04-22T22:08:14.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-04-22T22:08:14.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-04-22T22:08:14.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-04-22T22:08:14.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:14 smithi192 ceph-mon[22835]: osdmap e69: 8 total, 8 up, 8 in 2024-04-22T22:08:16.128 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:15 smithi174 ceph-mon[21825]: pgmap v80: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-22T22:08:16.129 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:15 smithi174 ceph-mon[21825]: osdmap e70: 8 total, 8 up, 8 in 2024-04-22T22:08:16.129 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:15 smithi174 ceph-mon[26730]: pgmap v80: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-22T22:08:16.129 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:15 smithi174 ceph-mon[26730]: osdmap e70: 8 total, 8 up, 8 in 2024-04-22T22:08:16.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:15 smithi192 ceph-mon[22835]: pgmap v80: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-22T22:08:16.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:15 smithi192 ceph-mon[22835]: osdmap e70: 8 total, 8 up, 8 in 2024-04-22T22:08:16.783 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled rgw.smpl update... 2024-04-22T22:08:17.538 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph osd pool create foo' 2024-04-22T22:08:17.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:17 smithi192 ceph-mon[22835]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:08:17.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:17 smithi192 ceph-mon[22835]: Saving service rgw.smpl spec with placement count:2 2024-04-22T22:08:17.904 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:17 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:17 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:17 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.ihnxdd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:08:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:17 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.ihnxdd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:08:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:17 smithi192 ceph-mon[22835]: pgmap v82: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:17 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:17 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:17.905 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:17 smithi192 ceph-mon[22835]: Deploying daemon rgw.foo.smithi192.ihnxdd on smithi192 2024-04-22T22:08:18.059 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[26730]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:08:18.059 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[26730]: Saving service rgw.smpl spec with placement count:2 2024-04-22T22:08:18.059 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:18.059 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:18.060 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.ihnxdd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:08:18.060 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.ihnxdd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:08:18.060 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[26730]: pgmap v82: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:18.060 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:18.060 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:18.060 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[26730]: Deploying daemon rgw.foo.smithi192.ihnxdd on smithi192 2024-04-22T22:08:18.060 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[21825]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:08:18.060 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[21825]: Saving service rgw.smpl spec with placement count:2 2024-04-22T22:08:18.060 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:18.060 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:18.060 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.ihnxdd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:08:18.060 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi192.ihnxdd", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:08:18.061 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[21825]: pgmap v82: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:18.061 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:18.061 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:18.061 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:17 smithi174 ceph-mon[21825]: Deploying daemon rgw.foo.smithi192.ihnxdd on smithi192 2024-04-22T22:08:20.030 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:19 smithi192 ceph-mon[22835]: pgmap v83: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:20.031 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:19 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:20.031 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:19 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:20.031 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:19 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:20.031 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:19 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1593128140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-22T22:08:20.031 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:19 smithi192 ceph-mon[22835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-22T22:08:20.031 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:19 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:20.031 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:19 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:20.031 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:19 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:20.031 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:19 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:20.031 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:19 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:20.218 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[21825]: pgmap v83: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:20.218 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:20.218 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:20.218 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:20.218 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1593128140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-22T22:08:20.218 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[21825]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-22T22:08:20.218 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:20.218 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:20.218 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:20.218 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:20.219 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:20.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[26730]: pgmap v83: 129 pgs: 4 peering, 125 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:20.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:20.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:20.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:20.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1593128140' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-22T22:08:20.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[26730]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-22T22:08:20.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:20.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:20.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:20.219 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:20.220 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:19 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:20.371 INFO:teuthology.orchestra.run.smithi174.stderr:pool 'foo' created 2024-04-22T22:08:21.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:21 smithi192 ceph-mon[22835]: Checking dashboard <-> RGW credentials 2024-04-22T22:08:21.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:21 smithi192 ceph-mon[22835]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:21.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:21 smithi192 ceph-mon[22835]: Checking dashboard <-> RGW credentials 2024-04-22T22:08:21.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:21 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:21.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:21 smithi192 ceph-mon[22835]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:21.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:21 smithi192 ceph-mon[22835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-22T22:08:21.095 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:21 smithi192 ceph-mon[22835]: osdmap e71: 8 total, 8 up, 8 in 2024-04-22T22:08:21.231 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[26730]: Checking dashboard <-> RGW credentials 2024-04-22T22:08:21.231 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[26730]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:21.232 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[26730]: Checking dashboard <-> RGW credentials 2024-04-22T22:08:21.232 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:21.232 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[26730]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:21.232 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[26730]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-22T22:08:21.232 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[26730]: osdmap e71: 8 total, 8 up, 8 in 2024-04-22T22:08:21.232 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[21825]: Checking dashboard <-> RGW credentials 2024-04-22T22:08:21.232 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[21825]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:21.232 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[21825]: Checking dashboard <-> RGW credentials 2024-04-22T22:08:21.232 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:21.232 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[21825]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:21.232 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[21825]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-22T22:08:21.232 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:21 smithi174 ceph-mon[21825]: osdmap e71: 8 total, 8 up, 8 in 2024-04-22T22:08:22.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:22 smithi192 ceph-mon[22835]: pgmap v85: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 4.7 KiB/s wr, 160 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T22:08:22.143 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:22 smithi192 ceph-mon[22835]: osdmap e72: 8 total, 8 up, 8 in 2024-04-22T22:08:22.199 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:22 smithi174 ceph-mon[21825]: pgmap v85: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 4.7 KiB/s wr, 160 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T22:08:22.199 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:22 smithi174 ceph-mon[21825]: osdmap e72: 8 total, 8 up, 8 in 2024-04-22T22:08:22.199 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:22 smithi174 ceph-mon[26730]: pgmap v85: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 66 KiB/s rd, 4.7 KiB/s wr, 160 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T22:08:22.199 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:22 smithi174 ceph-mon[26730]: osdmap e72: 8 total, 8 up, 8 in 2024-04-22T22:08:22.424 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'rbd pool init foo' 2024-04-22T22:08:22.449 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:22 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:08:22] "GET /metrics HTTP/1.1" 200 196925 "" "Prometheus/2.33.4" 2024-04-22T22:08:22.564 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:08:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:08:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:08:23.314 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:23 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:08:23] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:23.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:23 smithi192 ceph-mon[22835]: osdmap e73: 8 total, 8 up, 8 in 2024-04-22T22:08:23.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:23 smithi192 ceph-mon[22835]: pgmap v88: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.5 KiB/s wr, 186 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T22:08:23.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:23 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:23.402 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:23 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:23.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:23 smithi174 ceph-mon[21825]: osdmap e73: 8 total, 8 up, 8 in 2024-04-22T22:08:23.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:23 smithi174 ceph-mon[21825]: pgmap v88: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.5 KiB/s wr, 186 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T22:08:23.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:23 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:23.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:23 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:23.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:23 smithi174 ceph-mon[26730]: osdmap e73: 8 total, 8 up, 8 in 2024-04-22T22:08:23.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:23 smithi174 ceph-mon[26730]: pgmap v88: 161 pgs: 32 unknown, 129 active+clean; 584 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.5 KiB/s wr, 186 op/s; 0 B/s, 0 objects/s recovering 2024-04-22T22:08:23.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:23 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:23.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:23 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:24.780 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:24 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1167040008' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-22T22:08:24.780 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:24 smithi174 ceph-mon[21825]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-22T22:08:24.780 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:24 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1167040008' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-22T22:08:24.780 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:24 smithi174 ceph-mon[26730]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-22T22:08:24.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:24 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1167040008' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-22T22:08:24.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:24 smithi192 ceph-mon[22835]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-22T22:08:25.314 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:25 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:08:25] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:25.716 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:25 smithi192 ceph-mon[22835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-22T22:08:25.716 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:25 smithi192 ceph-mon[22835]: osdmap e74: 8 total, 8 up, 8 in 2024-04-22T22:08:25.716 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:25 smithi192 ceph-mon[22835]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 160 KiB/s rd, 682 B/s wr, 239 op/s 2024-04-22T22:08:25.716 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:25 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:25.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:25 smithi174 ceph-mon[21825]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-22T22:08:25.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:25 smithi174 ceph-mon[21825]: osdmap e74: 8 total, 8 up, 8 in 2024-04-22T22:08:25.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:25 smithi174 ceph-mon[21825]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 160 KiB/s rd, 682 B/s wr, 239 op/s 2024-04-22T22:08:25.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:25 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:25.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:25 smithi174 ceph-mon[26730]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-22T22:08:25.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:25 smithi174 ceph-mon[26730]: osdmap e74: 8 total, 8 up, 8 in 2024-04-22T22:08:25.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:25 smithi174 ceph-mon[26730]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 160 KiB/s rd, 682 B/s wr, 239 op/s 2024-04-22T22:08:25.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:25 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:26.314 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:26 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:08:26] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:26.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:26 smithi174 ceph-mon[26730]: osdmap e75: 8 total, 8 up, 8 in 2024-04-22T22:08:26.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:26 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:26.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:26 smithi192 ceph-mon[22835]: osdmap e75: 8 total, 8 up, 8 in 2024-04-22T22:08:26.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:26 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:26.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:26 smithi174 ceph-mon[21825]: osdmap e75: 8 total, 8 up, 8 in 2024-04-22T22:08:26.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:26 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:27.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:27 smithi192 ceph-mon[22835]: osdmap e76: 8 total, 8 up, 8 in 2024-04-22T22:08:27.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:27 smithi192 ceph-mon[22835]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 684 B/s wr, 240 op/s 2024-04-22T22:08:27.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:27 smithi174 ceph-mon[21825]: osdmap e76: 8 total, 8 up, 8 in 2024-04-22T22:08:27.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:27 smithi174 ceph-mon[21825]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 684 B/s wr, 240 op/s 2024-04-22T22:08:27.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:27 smithi174 ceph-mon[26730]: osdmap e76: 8 total, 8 up, 8 in 2024-04-22T22:08:27.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:27 smithi174 ceph-mon[26730]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 161 KiB/s rd, 684 B/s wr, 240 op/s 2024-04-22T22:08:27.815 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:27 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:08:27] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:27.928 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch apply iscsi foo u p' 2024-04-22T22:08:28.300 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:08:28] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:28.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: Saving service rgw.smpl spec with placement count:2 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi174.ggrvao", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi174.ggrvao", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: Deploying daemon rgw.smpl.smithi174.ggrvao on smithi174 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:28.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:28.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: Saving service rgw.smpl spec with placement count:2 2024-04-22T22:08:28.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi174.ggrvao", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:08:28.816 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi174.ggrvao", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:08:28.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:28.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: Deploying daemon rgw.smpl.smithi174.ggrvao on smithi174 2024-04-22T22:08:28.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.817 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:28 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.817 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:28.818 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:28.818 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.818 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: Saving service rgw.smpl spec with placement count:2 2024-04-22T22:08:28.818 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.818 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi174.ggrvao", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:08:28.818 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi174.ggrvao", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:08:28.818 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.818 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:28.818 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: Deploying daemon rgw.smpl.smithi174.ggrvao on smithi174 2024-04-22T22:08:28.818 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:28.818 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:28 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:29.678 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:29 smithi174 ceph-mon[21825]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 511 B/s wr, 179 op/s 2024-04-22T22:08:29.678 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:29 smithi174 ceph-mon[26730]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 511 B/s wr, 179 op/s 2024-04-22T22:08:29.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:29 smithi192 ceph-mon[22835]: pgmap v94: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 120 KiB/s rd, 511 B/s wr, 179 op/s 2024-04-22T22:08:31.333 INFO:teuthology.orchestra.run.smithi174.stdout:Scheduled iscsi.foo update... 2024-04-22T22:08:32.118 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:31 smithi174 ceph-mon[21825]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 323 B/s wr, 0 op/s 2024-04-22T22:08:32.118 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:31 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:32.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:31 smithi174 ceph-mon[26730]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 323 B/s wr, 0 op/s 2024-04-22T22:08:32.118 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:31 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:32.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:31 smithi192 ceph-mon[22835]: pgmap v95: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 323 B/s wr, 0 op/s 2024-04-22T22:08:32.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:31 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:32.519 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:32 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:08:32] "GET /metrics HTTP/1.1" 200 208128 "" "Prometheus/2.33.4" 2024-04-22T22:08:32.543 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'sleep 120' 2024-04-22T22:08:32.638 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:08:32 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:08:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:08:33.147 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[21825]: from='client.24733 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:08:33.147 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[21825]: Saving service iscsi.foo spec with placement count:1 2024-04-22T22:08:33.147 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[21825]: osdmap e77: 8 total, 8 up, 8 in 2024-04-22T22:08:33.147 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:33.147 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.nzyxdu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:08:33.147 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.nzyxdu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:08:33.147 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:33.148 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:33.148 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[21825]: Deploying daemon rgw.smpl.smithi192.nzyxdu on smithi192 2024-04-22T22:08:33.148 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[26730]: from='client.24733 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:08:33.148 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[26730]: Saving service iscsi.foo spec with placement count:1 2024-04-22T22:08:33.148 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[26730]: osdmap e77: 8 total, 8 up, 8 in 2024-04-22T22:08:33.148 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:33.148 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.nzyxdu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:08:33.148 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.nzyxdu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:08:33.148 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:33.148 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:33.148 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:33 smithi174 ceph-mon[26730]: Deploying daemon rgw.smpl.smithi192.nzyxdu on smithi192 2024-04-22T22:08:33.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:33 smithi192 ceph-mon[22835]: from='client.24733 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:08:33.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:33 smithi192 ceph-mon[22835]: Saving service iscsi.foo spec with placement count:1 2024-04-22T22:08:33.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:33 smithi192 ceph-mon[22835]: osdmap e77: 8 total, 8 up, 8 in 2024-04-22T22:08:33.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:33 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:33.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:33 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.nzyxdu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-22T22:08:33.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:33 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi192.nzyxdu", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-22T22:08:33.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:33 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:33.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:33 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:33.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:33 smithi192 ceph-mon[22835]: Deploying daemon rgw.smpl.smithi192.nzyxdu on smithi192 2024-04-22T22:08:34.338 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:34 smithi192 ceph-mon[22835]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 279 B/s wr, 0 op/s 2024-04-22T22:08:34.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:34 smithi174 ceph-mon[21825]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 279 B/s wr, 0 op/s 2024-04-22T22:08:34.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:34 smithi174 ceph-mon[26730]: pgmap v97: 161 pgs: 161 active+clean; 585 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 279 B/s wr, 0 op/s 2024-04-22T22:08:35.794 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:35.794 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:35.794 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:35.794 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[21825]: Checking dashboard <-> RGW credentials 2024-04-22T22:08:35.794 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:35.794 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[21825]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:35.795 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:35.795 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:35.795 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[21825]: Checking dashboard <-> RGW credentials 2024-04-22T22:08:35.795 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[21825]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 369 B/s wr, 121 op/s 2024-04-22T22:08:35.795 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:35.795 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[21825]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:35.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:35.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:35.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:35.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[26730]: Checking dashboard <-> RGW credentials 2024-04-22T22:08:35.795 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:35.796 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[26730]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:35.796 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:35.796 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:35.796 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[26730]: Checking dashboard <-> RGW credentials 2024-04-22T22:08:35.796 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[26730]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 369 B/s wr, 121 op/s 2024-04-22T22:08:35.796 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:35.796 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:35 smithi174 ceph-mon[26730]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:35.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:35.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:35.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:35.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:35 smithi192 ceph-mon[22835]: Checking dashboard <-> RGW credentials 2024-04-22T22:08:35.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:35.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:35 smithi192 ceph-mon[22835]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:35.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:35.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:35.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:35 smithi192 ceph-mon[22835]: Checking dashboard <-> RGW credentials 2024-04-22T22:08:35.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:35 smithi192 ceph-mon[22835]: pgmap v98: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 369 B/s wr, 121 op/s 2024-04-22T22:08:35.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:35 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:35.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:35 smithi192 ceph-mon[22835]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:38.256 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:38 smithi174 ceph-mon[21825]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 307 B/s wr, 100 op/s 2024-04-22T22:08:38.257 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:38 smithi174 ceph-mon[26730]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 307 B/s wr, 100 op/s 2024-04-22T22:08:38.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:38 smithi192 ceph-mon[22835]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 307 B/s wr, 100 op/s 2024-04-22T22:08:38.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:38 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:08:38] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:39.040 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:38 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:08:38] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:39.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:39 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:39.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:39 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:39.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:39 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:39.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:39 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:39.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:39 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:39.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:39 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:39.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:39 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:39.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:39 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:39.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:39 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:40.416 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:40 smithi174 ceph-mon[21825]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 307 B/s wr, 100 op/s 2024-04-22T22:08:40.417 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:40 smithi174 ceph-mon[26730]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 307 B/s wr, 100 op/s 2024-04-22T22:08:40.498 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:40 smithi192 ceph-mon[22835]: pgmap v100: 161 pgs: 161 active+clean; 585 KiB data, 70 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s rd, 307 B/s wr, 100 op/s 2024-04-22T22:08:40.814 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:08:40] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:41.314 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:40 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:08:40] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:41.590 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.590 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.591 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[21825]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 204 B/s wr, 207 op/s 2024-04-22T22:08:41.591 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.591 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.591 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:41.591 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:41.591 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.591 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[21825]: Checking pool "foo" exists for service iscsi.foo 2024-04-22T22:08:41.591 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi174.upousg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-22T22:08:41.591 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi174.upousg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-22T22:08:41.591 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:41.591 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[21825]: Deploying daemon iscsi.foo.smithi174.upousg on smithi174 2024-04-22T22:08:41.591 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.591 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[26730]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 204 B/s wr, 207 op/s 2024-04-22T22:08:41.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:41.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:41.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[26730]: Checking pool "foo" exists for service iscsi.foo 2024-04-22T22:08:41.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi174.upousg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-22T22:08:41.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi174.upousg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-22T22:08:41.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:41.592 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:41 smithi174 ceph-mon[26730]: Deploying daemon iscsi.foo.smithi174.upousg on smithi174 2024-04-22T22:08:41.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:41 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:41 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:41 smithi192 ceph-mon[22835]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 137 KiB/s rd, 204 B/s wr, 207 op/s 2024-04-22T22:08:41.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:41 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:41 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:41 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:41.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:41 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:41.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:41 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:41.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:41 smithi192 ceph-mon[22835]: Checking pool "foo" exists for service iscsi.foo 2024-04-22T22:08:41.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:41 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi174.upousg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-22T22:08:41.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:41 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi174.upousg", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-22T22:08:41.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:41 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:41.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:41 smithi192 ceph-mon[22835]: Deploying daemon iscsi.foo.smithi174.upousg on smithi174 2024-04-22T22:08:42.504 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:42 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:08:42] "GET /metrics HTTP/1.1" 200 208128 "" "Prometheus/2.33.4" 2024-04-22T22:08:42.814 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:08:42 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:08:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:08:43.757 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:08:43 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=warn ts=2024-04-22T22:08:43.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.174:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.174 because it doesn't contain any IP SANs" 2024-04-22T22:08:43.758 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:08:43 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=warn ts=2024-04-22T22:08:43.520Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.192:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.192 because it doesn't contain any IP SANs" 2024-04-22T22:08:44.104 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 189 B/s wr, 192 op/s 2024-04-22T22:08:44.104 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-22T22:08:44.104 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.104 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:44.104 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:44.104 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.105 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-22T22:08:44.105 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-22T22:08:44.105 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.105 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi174"}]: dispatch 2024-04-22T22:08:44.105 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.105 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:44.105 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:44.105 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.105 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 189 B/s wr, 192 op/s 2024-04-22T22:08:44.105 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-22T22:08:44.106 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.106 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:44.106 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:44.106 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.106 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-22T22:08:44.106 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-22T22:08:44.106 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.106 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi174"}]: dispatch 2024-04-22T22:08:44.106 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.106 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:44.106 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:44.106 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:44 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: pgmap v102: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 189 B/s wr, 192 op/s 2024-04-22T22:08:44.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-22T22:08:44.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:44.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:44.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-22T22:08:44.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-22T22:08:44.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi174"}]: dispatch 2024-04-22T22:08:44.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:44.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:44.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:44.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:44 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:45.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[21825]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:45.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[21825]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-22T22:08:45.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[21825]: Adding iSCSI gateway http://:@172.21.15.174:5000 to Dashboard 2024-04-22T22:08:45.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[21825]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-22T22:08:45.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[21825]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi174"}]: dispatch 2024-04-22T22:08:45.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[21825]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:45.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3798127431' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T22:08:45.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2796575011' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/381710589"}]: dispatch 2024-04-22T22:08:45.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:45 smithi192 ceph-mon[22835]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:45.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:45 smithi192 ceph-mon[22835]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-22T22:08:45.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:45 smithi192 ceph-mon[22835]: Adding iSCSI gateway http://:@172.21.15.174:5000 to Dashboard 2024-04-22T22:08:45.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:45 smithi192 ceph-mon[22835]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-22T22:08:45.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:45 smithi192 ceph-mon[22835]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi174"}]: dispatch 2024-04-22T22:08:45.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:45 smithi192 ceph-mon[22835]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:45.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:45 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3798127431' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T22:08:45.316 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:45 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2796575011' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/381710589"}]: dispatch 2024-04-22T22:08:45.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[26730]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:45.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[26730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-22T22:08:45.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[26730]: Adding iSCSI gateway http://:@172.21.15.174:5000 to Dashboard 2024-04-22T22:08:45.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[26730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-22T22:08:45.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[26730]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi174"}]: dispatch 2024-04-22T22:08:45.316 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[26730]: Metadata not up to date on all hosts. Skipping non agent specs 2024-04-22T22:08:45.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3798127431' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-22T22:08:45.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:45 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2796575011' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/381710589"}]: dispatch 2024-04-22T22:08:46.300 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:46 smithi174 ceph-mon[26730]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 170 B/s wr, 173 op/s 2024-04-22T22:08:46.300 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:46 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2796575011' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/381710589"}]': finished 2024-04-22T22:08:46.300 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:46 smithi174 ceph-mon[26730]: osdmap e78: 8 total, 8 up, 8 in 2024-04-22T22:08:46.300 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:46 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/762541880' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/3182264883"}]: dispatch 2024-04-22T22:08:46.300 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:46 smithi174 ceph-mon[26730]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/3182264883"}]: dispatch 2024-04-22T22:08:46.300 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:46 smithi174 ceph-mon[21825]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 170 B/s wr, 173 op/s 2024-04-22T22:08:46.301 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:46 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2796575011' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/381710589"}]': finished 2024-04-22T22:08:46.301 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:46 smithi174 ceph-mon[21825]: osdmap e78: 8 total, 8 up, 8 in 2024-04-22T22:08:46.301 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:46 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/762541880' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/3182264883"}]: dispatch 2024-04-22T22:08:46.301 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:46 smithi174 ceph-mon[21825]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/3182264883"}]: dispatch 2024-04-22T22:08:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:46 smithi192 ceph-mon[22835]: pgmap v103: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 114 KiB/s rd, 170 B/s wr, 173 op/s 2024-04-22T22:08:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:46 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2796575011' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/381710589"}]': finished 2024-04-22T22:08:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:46 smithi192 ceph-mon[22835]: osdmap e78: 8 total, 8 up, 8 in 2024-04-22T22:08:46.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:46 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/762541880' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/3182264883"}]: dispatch 2024-04-22T22:08:46.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:46 smithi192 ceph-mon[22835]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/3182264883"}]: dispatch 2024-04-22T22:08:47.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:47 smithi174 ceph-mon[21825]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/3182264883"}]': finished 2024-04-22T22:08:47.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:47 smithi174 ceph-mon[21825]: osdmap e79: 8 total, 8 up, 8 in 2024-04-22T22:08:47.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:47 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1573015676' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/3182264883"}]: dispatch 2024-04-22T22:08:47.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:47 smithi174 ceph-mon[21825]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/3182264883"}]: dispatch 2024-04-22T22:08:47.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:47 smithi174 ceph-mon[26730]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/3182264883"}]': finished 2024-04-22T22:08:47.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:47 smithi174 ceph-mon[26730]: osdmap e79: 8 total, 8 up, 8 in 2024-04-22T22:08:47.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:47 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1573015676' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/3182264883"}]: dispatch 2024-04-22T22:08:47.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:47 smithi174 ceph-mon[26730]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/3182264883"}]: dispatch 2024-04-22T22:08:47.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:47 smithi192 ceph-mon[22835]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/3182264883"}]': finished 2024-04-22T22:08:47.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:47 smithi192 ceph-mon[22835]: osdmap e79: 8 total, 8 up, 8 in 2024-04-22T22:08:47.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:47 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1573015676' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/3182264883"}]: dispatch 2024-04-22T22:08:47.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:47 smithi192 ceph-mon[22835]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/3182264883"}]: dispatch 2024-04-22T22:08:47.814 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:47 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:08:47] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:48.533 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:48 smithi174 ceph-mon[26730]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 127 B/s wr, 134 op/s 2024-04-22T22:08:48.533 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:48 smithi174 ceph-mon[26730]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/3182264883"}]': finished 2024-04-22T22:08:48.533 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:48 smithi174 ceph-mon[26730]: osdmap e80: 8 total, 8 up, 8 in 2024-04-22T22:08:48.533 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:48 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3998395231' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3592746357"}]: dispatch 2024-04-22T22:08:48.533 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:48 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:48.533 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:48 smithi174 ceph-mon[21825]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 127 B/s wr, 134 op/s 2024-04-22T22:08:48.534 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:48 smithi174 ceph-mon[21825]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/3182264883"}]': finished 2024-04-22T22:08:48.534 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:48 smithi174 ceph-mon[21825]: osdmap e80: 8 total, 8 up, 8 in 2024-04-22T22:08:48.534 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:48 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3998395231' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3592746357"}]: dispatch 2024-04-22T22:08:48.534 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:48 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:48.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:48 smithi192 ceph-mon[22835]: pgmap v106: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 89 KiB/s rd, 127 B/s wr, 134 op/s 2024-04-22T22:08:48.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:48 smithi192 ceph-mon[22835]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/3182264883"}]': finished 2024-04-22T22:08:48.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:48 smithi192 ceph-mon[22835]: osdmap e80: 8 total, 8 up, 8 in 2024-04-22T22:08:48.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:48 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3998395231' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3592746357"}]: dispatch 2024-04-22T22:08:48.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:48 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:49.358 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:49 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3998395231' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3592746357"}]': finished 2024-04-22T22:08:49.359 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:49 smithi174 ceph-mon[21825]: osdmap e81: 8 total, 8 up, 8 in 2024-04-22T22:08:49.359 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:49 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:49.359 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:49 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/345949634' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1494365845"}]: dispatch 2024-04-22T22:08:49.359 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:49 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3998395231' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3592746357"}]': finished 2024-04-22T22:08:49.359 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:49 smithi174 ceph-mon[26730]: osdmap e81: 8 total, 8 up, 8 in 2024-04-22T22:08:49.359 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:49 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:49.359 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:49 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/345949634' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1494365845"}]: dispatch 2024-04-22T22:08:49.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:49 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3998395231' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3592746357"}]': finished 2024-04-22T22:08:49.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:49 smithi192 ceph-mon[22835]: osdmap e81: 8 total, 8 up, 8 in 2024-04-22T22:08:49.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:49 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:49.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:49 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/345949634' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1494365845"}]: dispatch 2024-04-22T22:08:50.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:50 smithi174 ceph-mon[26730]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:50.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:50 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/345949634' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1494365845"}]': finished 2024-04-22T22:08:50.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:50 smithi174 ceph-mon[26730]: osdmap e82: 8 total, 8 up, 8 in 2024-04-22T22:08:50.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:50 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2804537570' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/2262747749"}]: dispatch 2024-04-22T22:08:50.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:50 smithi174 ceph-mon[21825]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:50.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:50 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/345949634' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1494365845"}]': finished 2024-04-22T22:08:50.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:50 smithi174 ceph-mon[21825]: osdmap e82: 8 total, 8 up, 8 in 2024-04-22T22:08:50.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:50 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2804537570' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/2262747749"}]: dispatch 2024-04-22T22:08:50.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:50 smithi192 ceph-mon[22835]: pgmap v109: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:50.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:50 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/345949634' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1494365845"}]': finished 2024-04-22T22:08:50.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:50 smithi192 ceph-mon[22835]: osdmap e82: 8 total, 8 up, 8 in 2024-04-22T22:08:50.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:50 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2804537570' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/2262747749"}]: dispatch 2024-04-22T22:08:50.814 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:50 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:08:50] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:51.517 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2804537570' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/2262747749"}]': finished 2024-04-22T22:08:51.517 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[21825]: osdmap e83: 8 total, 8 up, 8 in 2024-04-22T22:08:51.517 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:51.517 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:51.517 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3511544609' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/4157104731"}]: dispatch 2024-04-22T22:08:51.517 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[21825]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/4157104731"}]: dispatch 2024-04-22T22:08:51.517 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:51.517 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:51.517 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:51.518 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2804537570' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/2262747749"}]': finished 2024-04-22T22:08:51.518 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[26730]: osdmap e83: 8 total, 8 up, 8 in 2024-04-22T22:08:51.518 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:51.518 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:51.518 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3511544609' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/4157104731"}]: dispatch 2024-04-22T22:08:51.518 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[26730]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/4157104731"}]: dispatch 2024-04-22T22:08:51.518 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:51.518 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:51.518 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:51 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:51.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:51 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2804537570' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/2262747749"}]': finished 2024-04-22T22:08:51.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:51 smithi192 ceph-mon[22835]: osdmap e83: 8 total, 8 up, 8 in 2024-04-22T22:08:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:51 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:51 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:51 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3511544609' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/4157104731"}]: dispatch 2024-04-22T22:08:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:51 smithi192 ceph-mon[22835]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/4157104731"}]: dispatch 2024-04-22T22:08:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:51 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:08:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:51 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:08:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:51 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:52.466 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:08:52 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:08:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:08:52.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:52 smithi192 ceph-mon[22835]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-22T22:08:52.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:52 smithi192 ceph-mon[22835]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/4157104731"}]': finished 2024-04-22T22:08:52.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:52 smithi192 ceph-mon[22835]: osdmap e84: 8 total, 8 up, 8 in 2024-04-22T22:08:52.466 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:52 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/4206950647' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3519428216"}]: dispatch 2024-04-22T22:08:52.537 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:52 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:08:52] "GET /metrics HTTP/1.1" 200 215059 "" "Prometheus/2.33.4" 2024-04-22T22:08:52.537 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:52 smithi174 ceph-mon[21825]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-22T22:08:52.537 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:52 smithi174 ceph-mon[21825]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/4157104731"}]': finished 2024-04-22T22:08:52.537 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:52 smithi174 ceph-mon[21825]: osdmap e84: 8 total, 8 up, 8 in 2024-04-22T22:08:52.537 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:52 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/4206950647' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3519428216"}]: dispatch 2024-04-22T22:08:52.537 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:52 smithi174 ceph-mon[26730]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-22T22:08:52.538 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:52 smithi174 ceph-mon[26730]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/4157104731"}]': finished 2024-04-22T22:08:52.538 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:52 smithi174 ceph-mon[26730]: osdmap e84: 8 total, 8 up, 8 in 2024-04-22T22:08:52.538 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:52 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/4206950647' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3519428216"}]: dispatch 2024-04-22T22:08:53.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:53 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/4206950647' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3519428216"}]': finished 2024-04-22T22:08:53.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:53 smithi174 ceph-mon[21825]: osdmap e85: 8 total, 8 up, 8 in 2024-04-22T22:08:53.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:53 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2835012787' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/2262747749"}]: dispatch 2024-04-22T22:08:53.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:53 smithi174 ceph-mon[21825]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-22T22:08:53.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:53 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/4206950647' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3519428216"}]': finished 2024-04-22T22:08:53.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:53 smithi174 ceph-mon[26730]: osdmap e85: 8 total, 8 up, 8 in 2024-04-22T22:08:53.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:53 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2835012787' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/2262747749"}]: dispatch 2024-04-22T22:08:53.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:53 smithi174 ceph-mon[26730]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-22T22:08:53.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:08:53 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=error ts=2024-04-22T22:08:53.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.174:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.174 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.192:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.192 because it doesn't contain any IP SANs" 2024-04-22T22:08:53.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:08:53 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=warn ts=2024-04-22T22:08:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.174:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.174 because it doesn't contain any IP SANs" 2024-04-22T22:08:53.815 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:08:53 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=warn ts=2024-04-22T22:08:53.508Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.192:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.192 because it doesn't contain any IP SANs" 2024-04-22T22:08:53.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:53 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/4206950647' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3519428216"}]': finished 2024-04-22T22:08:53.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:53 smithi192 ceph-mon[22835]: osdmap e85: 8 total, 8 up, 8 in 2024-04-22T22:08:53.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:53 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2835012787' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/2262747749"}]: dispatch 2024-04-22T22:08:53.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:53 smithi192 ceph-mon[22835]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 3 op/s 2024-04-22T22:08:54.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:54 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2835012787' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/2262747749"}]': finished 2024-04-22T22:08:54.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:54 smithi174 ceph-mon[21825]: osdmap e86: 8 total, 8 up, 8 in 2024-04-22T22:08:54.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:54 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1568782353' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3967906832"}]: dispatch 2024-04-22T22:08:54.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:54 smithi174 ceph-mon[21825]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3967906832"}]: dispatch 2024-04-22T22:08:54.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:54 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:08:54.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:54 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2835012787' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/2262747749"}]': finished 2024-04-22T22:08:54.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:54 smithi174 ceph-mon[26730]: osdmap e86: 8 total, 8 up, 8 in 2024-04-22T22:08:54.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:54 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1568782353' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3967906832"}]: dispatch 2024-04-22T22:08:54.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:54 smithi174 ceph-mon[26730]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3967906832"}]: dispatch 2024-04-22T22:08:54.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:54 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:08:54.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:54 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2835012787' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/2262747749"}]': finished 2024-04-22T22:08:54.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:54 smithi192 ceph-mon[22835]: osdmap e86: 8 total, 8 up, 8 in 2024-04-22T22:08:54.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:54 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1568782353' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3967906832"}]: dispatch 2024-04-22T22:08:54.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:54 smithi192 ceph-mon[22835]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3967906832"}]: dispatch 2024-04-22T22:08:54.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:54 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:08:55.808 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:55 smithi192 ceph-mon[22835]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3967906832"}]': finished 2024-04-22T22:08:55.809 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:55 smithi192 ceph-mon[22835]: osdmap e87: 8 total, 8 up, 8 in 2024-04-22T22:08:55.809 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:55 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2484309293' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1131845225"}]: dispatch 2024-04-22T22:08:55.809 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:55 smithi192 ceph-mon[22835]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:08:55.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:55 smithi174 ceph-mon[21825]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3967906832"}]': finished 2024-04-22T22:08:55.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:55 smithi174 ceph-mon[21825]: osdmap e87: 8 total, 8 up, 8 in 2024-04-22T22:08:55.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:55 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2484309293' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1131845225"}]: dispatch 2024-04-22T22:08:55.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:55 smithi174 ceph-mon[21825]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:08:55.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:55 smithi174 ceph-mon[26730]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/3967906832"}]': finished 2024-04-22T22:08:55.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:55 smithi174 ceph-mon[26730]: osdmap e87: 8 total, 8 up, 8 in 2024-04-22T22:08:55.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:55 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2484309293' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1131845225"}]: dispatch 2024-04-22T22:08:55.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:55 smithi174 ceph-mon[26730]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:08:56.687 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:56 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2484309293' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1131845225"}]': finished 2024-04-22T22:08:56.687 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:56 smithi192 ceph-mon[22835]: osdmap e88: 8 total, 8 up, 8 in 2024-04-22T22:08:56.687 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:56 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/681793589' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1922808348"}]: dispatch 2024-04-22T22:08:56.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:56 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2484309293' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1131845225"}]': finished 2024-04-22T22:08:56.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:56 smithi174 ceph-mon[26730]: osdmap e88: 8 total, 8 up, 8 in 2024-04-22T22:08:56.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:56 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/681793589' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1922808348"}]: dispatch 2024-04-22T22:08:56.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:56 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2484309293' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1131845225"}]': finished 2024-04-22T22:08:56.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:56 smithi174 ceph-mon[21825]: osdmap e88: 8 total, 8 up, 8 in 2024-04-22T22:08:56.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:56 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/681793589' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1922808348"}]: dispatch 2024-04-22T22:08:57.814 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:08:57] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:08:57.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/681793589' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1922808348"}]': finished 2024-04-22T22:08:57.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[21825]: osdmap e89: 8 total, 8 up, 8 in 2024-04-22T22:08:57.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/24543953' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/2470543672"}]: dispatch 2024-04-22T22:08:57.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[21825]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/2470543672"}]: dispatch 2024-04-22T22:08:57.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[21825]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:08:57.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:57.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:57.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/681793589' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1922808348"}]': finished 2024-04-22T22:08:57.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[26730]: osdmap e89: 8 total, 8 up, 8 in 2024-04-22T22:08:57.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/24543953' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/2470543672"}]: dispatch 2024-04-22T22:08:57.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[26730]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/2470543672"}]: dispatch 2024-04-22T22:08:57.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[26730]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:08:57.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:57.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:57 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:57.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:57 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/681793589' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1922808348"}]': finished 2024-04-22T22:08:57.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:57 smithi192 ceph-mon[22835]: osdmap e89: 8 total, 8 up, 8 in 2024-04-22T22:08:57.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:57 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/24543953' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/2470543672"}]: dispatch 2024-04-22T22:08:57.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:57 smithi192 ceph-mon[22835]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/2470543672"}]: dispatch 2024-04-22T22:08:57.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:57 smithi192 ceph-mon[22835]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:08:57.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:57 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:57.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:57 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:08:58.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:58 smithi174 ceph-mon[21825]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/2470543672"}]': finished 2024-04-22T22:08:58.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:58 smithi174 ceph-mon[21825]: osdmap e90: 8 total, 8 up, 8 in 2024-04-22T22:08:58.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:58 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/162378381' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1468603511"}]: dispatch 2024-04-22T22:08:58.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:58 smithi174 ceph-mon[26730]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/2470543672"}]': finished 2024-04-22T22:08:58.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:58 smithi174 ceph-mon[26730]: osdmap e90: 8 total, 8 up, 8 in 2024-04-22T22:08:58.815 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:58 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/162378381' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1468603511"}]: dispatch 2024-04-22T22:08:58.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:58 smithi192 ceph-mon[22835]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/2470543672"}]': finished 2024-04-22T22:08:58.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:58 smithi192 ceph-mon[22835]: osdmap e90: 8 total, 8 up, 8 in 2024-04-22T22:08:58.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:58 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/162378381' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1468603511"}]: dispatch 2024-04-22T22:08:59.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:59 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/162378381' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1468603511"}]': finished 2024-04-22T22:08:59.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:59 smithi174 ceph-mon[26730]: osdmap e91: 8 total, 8 up, 8 in 2024-04-22T22:08:59.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:59 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1356371906' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/4261879208"}]: dispatch 2024-04-22T22:08:59.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:08:59 smithi174 ceph-mon[26730]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:59.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:59 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/162378381' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1468603511"}]': finished 2024-04-22T22:08:59.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:59 smithi174 ceph-mon[21825]: osdmap e91: 8 total, 8 up, 8 in 2024-04-22T22:08:59.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:59 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1356371906' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/4261879208"}]: dispatch 2024-04-22T22:08:59.815 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:08:59 smithi174 ceph-mon[21825]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:08:59.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:59 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/162378381' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:0/1468603511"}]': finished 2024-04-22T22:08:59.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:59 smithi192 ceph-mon[22835]: osdmap e91: 8 total, 8 up, 8 in 2024-04-22T22:08:59.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:59 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1356371906' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/4261879208"}]: dispatch 2024-04-22T22:08:59.816 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:08:59 smithi192 ceph-mon[22835]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail 2024-04-22T22:09:01.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:00 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/1356371906' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/4261879208"}]': finished 2024-04-22T22:09:01.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:00 smithi192 ceph-mon[22835]: osdmap e92: 8 total, 8 up, 8 in 2024-04-22T22:09:01.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:00 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/985816622' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/4261879208"}]: dispatch 2024-04-22T22:09:01.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:00 smithi192 ceph-mon[22835]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/4261879208"}]: dispatch 2024-04-22T22:09:01.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:00 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/1356371906' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/4261879208"}]': finished 2024-04-22T22:09:01.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:00 smithi174 ceph-mon[21825]: osdmap e92: 8 total, 8 up, 8 in 2024-04-22T22:09:01.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:00 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/985816622' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/4261879208"}]: dispatch 2024-04-22T22:09:01.065 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:00 smithi174 ceph-mon[21825]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/4261879208"}]: dispatch 2024-04-22T22:09:01.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:00 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/1356371906' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6800/4261879208"}]': finished 2024-04-22T22:09:01.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:00 smithi174 ceph-mon[26730]: osdmap e92: 8 total, 8 up, 8 in 2024-04-22T22:09:01.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:00 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/985816622' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/4261879208"}]: dispatch 2024-04-22T22:09:01.066 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:00 smithi174 ceph-mon[26730]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/4261879208"}]: dispatch 2024-04-22T22:09:02.064 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:01 smithi174 ceph-mon[21825]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/4261879208"}]': finished 2024-04-22T22:09:02.064 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:01 smithi174 ceph-mon[21825]: osdmap e93: 8 total, 8 up, 8 in 2024-04-22T22:09:02.064 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:01 smithi174 ceph-mon[21825]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:02.065 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:01 smithi174 ceph-mon[26730]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/4261879208"}]': finished 2024-04-22T22:09:02.065 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:01 smithi174 ceph-mon[26730]: osdmap e93: 8 total, 8 up, 8 in 2024-04-22T22:09:02.065 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:01 smithi174 ceph-mon[26730]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:02.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:01 smithi192 ceph-mon[22835]: from='client.? ' entity='client.iscsi.foo.smithi174.upousg' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.174:6801/4261879208"}]': finished 2024-04-22T22:09:02.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:01 smithi192 ceph-mon[22835]: osdmap e93: 8 total, 8 up, 8 in 2024-04-22T22:09:02.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:01 smithi192 ceph-mon[22835]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:02.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:09:02 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:09:02] "GET /metrics HTTP/1.1" 200 215064 "" "Prometheus/2.33.4" 2024-04-22T22:09:02.814 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:09:02 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:09:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:09:03.814 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:09:03 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=error ts=2024-04-22T22:09:03.507Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.174:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.174 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.192:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.192 because it doesn't contain any IP SANs" 2024-04-22T22:09:03.814 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:09:03 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=warn ts=2024-04-22T22:09:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.174:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.174 because it doesn't contain any IP SANs" 2024-04-22T22:09:03.814 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:09:03 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=warn ts=2024-04-22T22:09:03.509Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.192:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.192 because it doesn't contain any IP SANs" 2024-04-22T22:09:04.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:04 smithi174 ceph-mon[21825]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-04-22T22:09:04.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:04 smithi174 ceph-mon[26730]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-04-22T22:09:04.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:04 smithi192 ceph-mon[22835]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 962 B/s rd, 0 op/s 2024-04-22T22:09:04.802 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:09:04 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:09:04] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:09:05.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:05 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:05.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:05 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:05.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:05 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:05.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:05 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:05.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:05 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:05.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:05 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:05.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:05 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:05.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:05 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:05.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:05 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:06.134 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:06 smithi174 ceph-mon[21825]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-22T22:09:06.134 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:06 smithi174 ceph-mon[26730]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-22T22:09:06.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:06 smithi192 ceph-mon[22835]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-04-22T22:09:08.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:08 smithi192 ceph-mon[22835]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:08.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:08 smithi174 ceph-mon[21825]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:08.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:08 smithi174 ceph-mon[26730]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:10.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:10 smithi174 ceph-mon[21825]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 549 B/s rd, 0 op/s 2024-04-22T22:09:10.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:10 smithi174 ceph-mon[26730]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 549 B/s rd, 0 op/s 2024-04-22T22:09:10.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:10 smithi192 ceph-mon[22835]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 549 B/s rd, 0 op/s 2024-04-22T22:09:12.385 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:12 smithi192 ceph-mon[22835]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2024-04-22T22:09:12.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:12 smithi174 ceph-mon[21825]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2024-04-22T22:09:12.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:12 smithi174 ceph-mon[26730]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 993 B/s rd, 0 op/s 2024-04-22T22:09:12.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:09:12 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:09:12] "GET /metrics HTTP/1.1" 200 215064 "" "Prometheus/2.33.4" 2024-04-22T22:09:12.814 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:09:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:09:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:09:13.814 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:09:13 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=error ts=2024-04-22T22:09:13.508Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://172.21.15.174:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.174 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://172.21.15.192:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.192 because it doesn't contain any IP SANs" 2024-04-22T22:09:14.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:14 smithi174 ceph-mon[21825]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:14.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:09:14.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:09:14.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:14 smithi174 ceph-mon[26730]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:14.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:09:14.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:09:14.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:14 smithi192 ceph-mon[22835]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:14.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:09:14.566 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:09:15.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:15 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:15.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:15 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:15.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:15 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:16.383 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:16 smithi192 ceph-mon[22835]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:16.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:16 smithi174 ceph-mon[21825]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:16.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:16 smithi174 ceph-mon[26730]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:18.064 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:09:17 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:09:17] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:09:18.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:18 smithi192 ceph-mon[22835]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:18.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:18 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:18.528 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:18 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:18.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:18 smithi174 ceph-mon[21825]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:18.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:18 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:18.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:18 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:18.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:18 smithi174 ceph-mon[26730]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:18.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:18 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:18.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:18 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:20.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:20 smithi174 ceph-mon[21825]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:20.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:20 smithi174 ceph-mon[26730]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:20.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:20 smithi192 ceph-mon[22835]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:22.366 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:22 smithi192 ceph-mon[22835]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:22.469 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:22 smithi174 ceph-mon[21825]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:22.469 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:22 smithi174 ceph-mon[26730]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:22.469 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:09:22 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:09:22] "GET /metrics HTTP/1.1" 200 215053 "" "Prometheus/2.33.4" 2024-04-22T22:09:22.814 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:09:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:09:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:09:24.530 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:24 smithi192 ceph-mon[22835]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-22T22:09:24.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:24 smithi174 ceph-mon[26730]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-22T22:09:24.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:24 smithi174 ceph-mon[21825]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-04-22T22:09:25.101 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:09:24 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:09:24] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:09:25.503 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:25 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:25.503 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:25 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:25.503 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:25 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:25.503 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:25 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:25.504 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:25 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:25.504 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:25 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:25.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:25 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:25.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:25 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:25.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:25 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:26.494 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:26 smithi174 ceph-mon[21825]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:26.495 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:26 smithi174 ceph-mon[26730]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:26.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:26 smithi192 ceph-mon[22835]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:28.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:28 smithi174 ceph-mon[21825]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:28.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:28 smithi174 ceph-mon[26730]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:28.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:28 smithi192 ceph-mon[22835]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:30.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:30 smithi174 ceph-mon[21825]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:30.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:30 smithi192 ceph-mon[22835]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:30.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:30 smithi174 ceph-mon[26730]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:32.385 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:32 smithi192 ceph-mon[22835]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:32.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:32 smithi174 ceph-mon[21825]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:32.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:32 smithi174 ceph-mon[26730]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:32.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:09:32 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:09:32] "GET /metrics HTTP/1.1" 200 214999 "" "Prometheus/2.33.4" 2024-04-22T22:09:32.814 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:09:32 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:09:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:09:34.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:34 smithi192 ceph-mon[22835]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:34.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:34 smithi174 ceph-mon[21825]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:34.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:34 smithi174 ceph-mon[26730]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:35.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:35 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:35.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:35 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:35.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:35 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:36.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:36 smithi174 ceph-mon[21825]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:36.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:36 smithi174 ceph-mon[26730]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:36.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:36 smithi192 ceph-mon[22835]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:38.064 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:09:37 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:09:37] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:09:38.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:38 smithi192 ceph-mon[22835]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:38.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:38 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:38.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:38 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:38.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:38 smithi174 ceph-mon[21825]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:38.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:38 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:38.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:38 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:38.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:38 smithi174 ceph-mon[26730]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:38.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:38 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:38.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:38 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:40.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:40 smithi174 ceph-mon[21825]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:40.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:40 smithi174 ceph-mon[26730]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:40.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:40 smithi192 ceph-mon[22835]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:42.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:42 smithi174 ceph-mon[21825]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:42.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:42 smithi174 ceph-mon[26730]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:42.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:09:42 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:09:42] "GET /metrics HTTP/1.1" 200 214999 "" "Prometheus/2.33.4" 2024-04-22T22:09:42.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:42 smithi192 ceph-mon[22835]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:42.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:09:42 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:09:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:09:44.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:44 smithi174 ceph-mon[21825]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:44.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:44 smithi174 ceph-mon[26730]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:44.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:44 smithi192 ceph-mon[22835]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:45.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:45 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:45.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:45.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:45.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:09:44 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:09:44] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:09:45.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:45 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:45.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:45.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:45.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:45 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:45.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:45.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:46.430 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:46 smithi174 ceph-mon[21825]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:46.430 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:46 smithi174 ceph-mon[26730]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:46.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:46 smithi192 ceph-mon[22835]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:48.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:48 smithi174 ceph-mon[21825]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:48.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:48 smithi174 ceph-mon[26730]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:48.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:48 smithi192 ceph-mon[22835]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:50.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:50 smithi174 ceph-mon[21825]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:50.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:50 smithi174 ceph-mon[26730]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:50.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:50 smithi192 ceph-mon[22835]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:51.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:51 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:09:51.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:51 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:09:51.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:51 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:51.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:51 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:09:51.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:51 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:09:51.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:51 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:51 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:09:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:51 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:09:51.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:51 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:52.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:52 smithi192 ceph-mon[22835]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:52.465 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:52 smithi174 ceph-mon[21825]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:52.466 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:09:52 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:09:52] "GET /metrics HTTP/1.1" 200 214971 "" "Prometheus/2.33.4" 2024-04-22T22:09:52.466 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:52 smithi174 ceph-mon[26730]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:52.814 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:09:52 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:09:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:09:54.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:54 smithi192 ceph-mon[22835]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:54.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:54 smithi174 ceph-mon[21825]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:54.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:54 smithi174 ceph-mon[26730]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:55.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:55 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:55.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:55 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:55.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:55 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:09:56.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:56 smithi192 ceph-mon[22835]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:56.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:56 smithi174 ceph-mon[21825]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:56.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:56 smithi174 ceph-mon[26730]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:09:58.064 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:09:57 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:09:57] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:09:58.422 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:58 smithi192 ceph-mon[22835]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:58.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:58 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:58.423 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:09:58 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:58.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:58 smithi174 ceph-mon[21825]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:58.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:58 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:58.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:09:58 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:58.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:58 smithi174 ceph-mon[26730]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:09:58.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:58 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:09:58.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:09:58 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:00.493 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:00 smithi192 ceph-mon[22835]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:00.493 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:00 smithi192 ceph-mon[22835]: overall HEALTH_OK 2024-04-22T22:10:00.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:00 smithi174 ceph-mon[21825]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:00.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:00 smithi174 ceph-mon[21825]: overall HEALTH_OK 2024-04-22T22:10:00.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:00 smithi174 ceph-mon[26730]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:00.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:00 smithi174 ceph-mon[26730]: overall HEALTH_OK 2024-04-22T22:10:02.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:02 smithi174 ceph-mon[21825]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:02.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:02 smithi174 ceph-mon[26730]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:02.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:10:02 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:10:02] "GET /metrics HTTP/1.1" 200 214977 "" "Prometheus/2.33.4" 2024-04-22T22:10:02.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:02 smithi192 ceph-mon[22835]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:02.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:10:02 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:10:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:10:04.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:04 smithi174 ceph-mon[21825]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:04.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:04 smithi174 ceph-mon[26730]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:04.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:04 smithi192 ceph-mon[22835]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:05.314 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:10:05 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:10:05] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:10:05.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:05 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:05.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:05 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:05.315 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:05 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:05.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:05 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:05.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:05 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:05.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:05 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:05.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:05 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:05.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:05 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:05.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:05 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:06.450 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:06 smithi174 ceph-mon[26730]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:06.451 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:06 smithi174 ceph-mon[21825]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:06.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:06 smithi192 ceph-mon[22835]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:08.384 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:08 smithi174 ceph-mon[21825]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:08.385 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:08 smithi174 ceph-mon[26730]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:08.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:08 smithi192 ceph-mon[22835]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:10.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:10 smithi192 ceph-mon[22835]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:10.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:10 smithi174 ceph-mon[21825]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:10.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:10 smithi174 ceph-mon[26730]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:12.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:12 smithi192 ceph-mon[22835]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:12.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:12 smithi174 ceph-mon[21825]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:12.564 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:10:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:10:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:10:12.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:12 smithi174 ceph-mon[26730]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:12.565 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:10:12 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:10:12] "GET /metrics HTTP/1.1" 200 214977 "" "Prometheus/2.33.4" 2024-04-22T22:10:13.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:10:13.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:10:13.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:10:13.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:10:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:10:13.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:10:14.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:14 smithi192 ceph-mon[22835]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:14.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:14 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:14.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:14 smithi174 ceph-mon[21825]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:14.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:14 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:14.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:14 smithi174 ceph-mon[26730]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:14.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:14 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:15.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:15 smithi192 ceph-mon[22835]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:15.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:15 smithi174 ceph-mon[21825]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:15.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:15 smithi174 ceph-mon[26730]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:18.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:18 smithi192 ceph-mon[22835]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:18.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:18 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:18.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:18 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:18.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:18 smithi174 ceph-mon[21825]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:18.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:18 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:18.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:18 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:18.315 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:10:17 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:10:17] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:10:18.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:18 smithi174 ceph-mon[26730]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:18.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:18 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:18.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:18 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:20.484 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:20 smithi192 ceph-mon[22835]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:20.495 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:20 smithi174 ceph-mon[21825]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:20.496 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:20 smithi174 ceph-mon[26730]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:22.235 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:22 smithi192 ceph-mon[22835]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:22.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:22 smithi174 ceph-mon[21825]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:22.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:22 smithi174 ceph-mon[26730]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:22.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:10:22 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:10:22] "GET /metrics HTTP/1.1" 200 214998 "" "Prometheus/2.33.4" 2024-04-22T22:10:22.565 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:10:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:10:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:10:24.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:24 smithi192 ceph-mon[22835]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:24.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:24 smithi174 ceph-mon[21825]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:24.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:24 smithi174 ceph-mon[26730]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:25.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:10:25 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:10:25] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:10:26.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:26 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:26.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:26 smithi192 ceph-mon[22835]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:26.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:26 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:26.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:26 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:26.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:26 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:26.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:26 smithi174 ceph-mon[21825]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:26.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:26 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:26.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:26 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:26.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:26 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:26.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:26 smithi174 ceph-mon[26730]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:26.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:26 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:26.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:26 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:28.131 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:28 smithi174 ceph-mon[21825]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:28.563 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:28 smithi174 ceph-mon[26730]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:28.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:28 smithi192 ceph-mon[22835]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:30.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:30 smithi192 ceph-mon[22835]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:30.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:30 smithi174 ceph-mon[21825]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:30.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:30 smithi174 ceph-mon[26730]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:32.432 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:10:32 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:10:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:10:32.432 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:32 smithi192 ceph-mon[22835]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:32.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:32 smithi174 ceph-mon[26730]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:32.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:32 smithi174 ceph-mon[21825]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:32.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:10:32 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:10:32] "GET /metrics HTTP/1.1" 200 215008 "" "Prometheus/2.33.4" 2024-04-22T22:10:34.486 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:34 smithi174 ceph-mon[21825]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:34.486 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:34 smithi174 ceph-mon[26730]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:34.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:34 smithi192 ceph-mon[22835]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:34.780 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-22T22:10:36.408 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:36 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:36.408 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:36 smithi174 ceph-mon[21825]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:36.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:36 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:36.408 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:36 smithi174 ceph-mon[26730]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:36.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:36 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:36.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:36 smithi192 ceph-mon[22835]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:37.540 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-22T22:10:38.146 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:10:38 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:10:38] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:10:38.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:38 smithi192 ceph-mon[22835]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:38.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:38 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:38.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:38 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:38.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:38 smithi174 ceph-mon[26730]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:38.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:38 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:38.566 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:38 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:38.567 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:38 smithi174 ceph-mon[21825]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:38.567 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:38 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:38.567 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:38 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:40.055 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-22T22:10:40.441 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:40 smithi192 ceph-mon[22835]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:40.467 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:40 smithi174 ceph-mon[21825]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:40.467 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:40 smithi174 ceph-mon[26730]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:42.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:42 smithi174 ceph-mon[21825]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:42.314 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:10:42 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:10:42] "GET /metrics HTTP/1.1" 200 215008 "" "Prometheus/2.33.4" 2024-04-22T22:10:42.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:42 smithi174 ceph-mon[26730]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:42.315 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:42 smithi192 ceph-mon[22835]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:42.659 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-22T22:10:42.814 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:10:42 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:10:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:10:44.192 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:44 smithi174 ceph-mon[21825]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:44.192 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:44 smithi174 ceph-mon[26730]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:44.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:44 smithi192 ceph-mon[22835]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:44.597 INFO:teuthology.orchestra.run.smithi174.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:10:45.314 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:10:45.035+0000 7f9eeacba700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T22:10:45.314 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:10:45] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:10:45.399 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-22T22:10:45.408 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi174.front.sepia.ceph.com 2024-04-22T22:10:45.409 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-22T22:10:45.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:45.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:10:45.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:10:45.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:45.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:10:45.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:10:45.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:45.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:10:45.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:45.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T22:10:45.710 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:45.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:45.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:10:45.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:10:45.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:45.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:45.711 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:45.711 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:45.711 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:10:45.711 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:10:45.711 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:45.711 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:10:45.711 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:10:45.711 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:45.712 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:10:45.712 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:45.712 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T22:10:45.712 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:45.712 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:45.712 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:10:45.712 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:10:45.712 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:45.712 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:45.712 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:46.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:46.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: from='client.15078 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:10:46.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:10:46.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:46.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:10:46.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:10:46.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:46.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:10:46.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:10:46.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T22:10:46.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:46.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:46.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:10:46.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:10:46.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:46.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:46.065 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:46.749 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:46 smithi174 ceph-mon[21825]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-22T22:10:46.749 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:46 smithi174 ceph-mon[26730]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-22T22:10:47.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:46 smithi192 ceph-mon[22835]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-22T22:10:47.419 INFO:teuthology.orchestra.run.smithi174.stdout:true 2024-04-22T22:10:47.722 INFO:teuthology.orchestra.run.smithi174.stdout:"Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-22T22:10:48.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:48 smithi174 ceph-mon[21825]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:48.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:48 smithi174 ceph-mon[26730]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:48.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:48 smithi192 ceph-mon[22835]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:49.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:49 smithi174 ceph-mon[21825]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:10:49.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:49 smithi174 ceph-mon[21825]: from='client.24994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:10:49.317 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:49 smithi174 ceph-mon[26730]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:10:49.318 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:49 smithi174 ceph-mon[26730]: from='client.24994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:10:49.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:49 smithi192 ceph-mon[22835]: from='client.15084 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:10:49.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:49 smithi192 ceph-mon[22835]: from='client.24994 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:10:49.629 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch ps' 2024-04-22T22:10:50.261 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:50 smithi174 ceph-mon[21825]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:50.261 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:50 smithi174 ceph-mon[26730]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:50.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:50 smithi192 ceph-mon[22835]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:52.213 INFO:teuthology.orchestra.run.smithi174.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T22:10:52.214 INFO:teuthology.orchestra.run.smithi174.stdout:agent.smithi174 smithi174 running 6s ago 8m - - 2024-04-22T22:10:52.214 INFO:teuthology.orchestra.run.smithi174.stdout:agent.smithi192 smithi192 running 14s ago 8m - - 2024-04-22T22:10:52.214 INFO:teuthology.orchestra.run.smithi174.stdout:alertmanager.a smithi174 *:9093,9094 running (3m) 6s ago 3m 26.8M - ba2b418f427c 1e9de7a331d8 2024-04-22T22:10:52.214 INFO:teuthology.orchestra.run.smithi174.stdout:grafana.a smithi192 *:3000 running (3m) 14s ago 3m 57.1M - 8.3.5 dad864ee21e9 4787c8784d2e 2024-04-22T22:10:52.214 INFO:teuthology.orchestra.run.smithi174.stdout:iscsi.foo.smithi174.upousg smithi174 running (2m) 6s ago 2m 48.5M - 3.5 e1d6a67b021e cc36500f1ff2 2024-04-22T22:10:52.214 INFO:teuthology.orchestra.run.smithi174.stdout:mgr.x smithi192 *:8443 running (7m) 14s ago 7m 412M - 17.2.0 e1d6a67b021e bd398d520007 2024-04-22T22:10:52.214 INFO:teuthology.orchestra.run.smithi174.stdout:mgr.y smithi174 *:9283 running (9m) 6s ago 9m 464M - 17.2.0 e1d6a67b021e 739b59b845dc 2024-04-22T22:10:52.214 INFO:teuthology.orchestra.run.smithi174.stdout:mon.a smithi174 running (9m) 6s ago 9m 52.5M 2048M 17.2.0 e1d6a67b021e d85a5efa7dc5 2024-04-22T22:10:52.214 INFO:teuthology.orchestra.run.smithi174.stdout:mon.b smithi192 running (8m) 14s ago 8m 40.9M 2048M 17.2.0 e1d6a67b021e 4d13e5913d29 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:mon.c smithi174 running (8m) 6s ago 8m 40.7M 2048M 17.2.0 e1d6a67b021e 260b4acfe3f8 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:node-exporter.a smithi174 *:9100 running (4m) 6s ago 4m 16.7M - 1dbe0e931976 5e1c68495a2c 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:node-exporter.b smithi192 *:9100 running (4m) 14s ago 4m 16.1M - 1dbe0e931976 71966acd94b1 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:osd.0 smithi174 running (7m) 6s ago 7m 50.1M 4096M 17.2.0 e1d6a67b021e b5aa9f2bad5d 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:osd.1 smithi174 running (7m) 6s ago 7m 54.0M 4096M 17.2.0 e1d6a67b021e 926b9f070a23 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:osd.2 smithi174 running (6m) 6s ago 6m 45.7M 4096M 17.2.0 e1d6a67b021e 830775ff8596 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:osd.3 smithi174 running (6m) 6s ago 6m 49.0M 4096M 17.2.0 e1d6a67b021e 0922e1ebb4ed 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:osd.4 smithi192 running (6m) 14s ago 6m 51.4M 4096M 17.2.0 e1d6a67b021e c77c665e8b26 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:osd.5 smithi192 running (5m) 14s ago 5m 46.7M 4096M 17.2.0 e1d6a67b021e 67848ad026e0 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:osd.6 smithi192 running (5m) 14s ago 5m 45.9M 4096M 17.2.0 e1d6a67b021e d75a88722468 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:osd.7 smithi192 running (5m) 14s ago 5m 51.7M 4096M 17.2.0 e1d6a67b021e a4170e31930d 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:prometheus.a smithi192 *:9095 running (2m) 14s ago 4m 63.6M - 514e6a882f6e dfa924a46ee2 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:rgw.foo.smithi174.eqkeoc smithi174 *:8000 running (2m) 6s ago 2m 69.7M - 17.2.0 e1d6a67b021e 82a13f9621e8 2024-04-22T22:10:52.215 INFO:teuthology.orchestra.run.smithi174.stdout:rgw.foo.smithi192.ihnxdd smithi192 *:8000 running (2m) 14s ago 2m 71.0M - 17.2.0 e1d6a67b021e 1ad669d97de3 2024-04-22T22:10:52.216 INFO:teuthology.orchestra.run.smithi174.stdout:rgw.smpl.smithi174.ggrvao smithi174 *:80 running (2m) 6s ago 2m 69.3M - 17.2.0 e1d6a67b021e 5b3499dd9c27 2024-04-22T22:10:52.216 INFO:teuthology.orchestra.run.smithi174.stdout:rgw.smpl.smithi192.nzyxdu smithi192 *:80 running (2m) 14s ago 2m 70.4M - 17.2.0 e1d6a67b021e bcc184444db6 2024-04-22T22:10:52.321 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:52 smithi174 ceph-mon[21825]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:52.322 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:52 smithi174 ceph-mon[26730]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:52.322 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:10:52 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:10:52] "GET /metrics HTTP/1.1" 200 214999 "" "Prometheus/2.33.4" 2024-04-22T22:10:52.366 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:52 smithi192 ceph-mon[22835]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:52.666 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:10:52 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:10:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:10:52.939 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions' 2024-04-22T22:10:54.374 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:54 smithi174 ceph-mon[26730]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:10:54.375 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:54 smithi174 ceph-mon[26730]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:54.375 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:54 smithi174 ceph-mon[21825]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:10:54.375 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:54 smithi174 ceph-mon[21825]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:54.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:54 smithi192 ceph-mon[22835]: from='client.25000 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:10:54.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:54 smithi192 ceph-mon[22835]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:55.106 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-22T22:10:55.106 INFO:teuthology.orchestra.run.smithi174.stdout: "mon": { 2024-04-22T22:10:55.106 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-22T22:10:55.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:10:55.106 INFO:teuthology.orchestra.run.smithi174.stdout: "mgr": { 2024-04-22T22:10:55.106 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T22:10:55.106 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:10:55.106 INFO:teuthology.orchestra.run.smithi174.stdout: "osd": { 2024-04-22T22:10:55.107 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-22T22:10:55.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:10:55.107 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": {}, 2024-04-22T22:10:55.107 INFO:teuthology.orchestra.run.smithi174.stdout: "rgw": { 2024-04-22T22:10:55.107 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-22T22:10:55.107 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:10:55.107 INFO:teuthology.orchestra.run.smithi174.stdout: "overall": { 2024-04-22T22:10:55.107 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-22T22:10:55.107 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-22T22:10:55.107 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-22T22:10:55.748 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-22T22:10:56.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:56 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:56.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:56 smithi174 ceph-mon[21825]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:56.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:56 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/3149017726' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:10:56.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:56 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:56.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:56 smithi174 ceph-mon[26730]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:56.315 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:56 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/3149017726' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:10:56.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:56 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:10:56.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:56 smithi192 ceph-mon[22835]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s 2024-04-22T22:10:56.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:56 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/3149017726' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:10:57.583 INFO:teuthology.orchestra.run.smithi174.stdout:wait for servicemap items w/ changing names to refresh 2024-04-22T22:10:58.354 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:58 smithi174 ceph-mon[21825]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:58.354 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:58 smithi174 ceph-mon[26730]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:58.355 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:10:58 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:10:58] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:10:58.376 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'sleep 60' 2024-04-22T22:10:58.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:58 smithi192 ceph-mon[22835]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:10:59.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:59 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:59.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:59 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:59.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:10:59 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:59.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:59 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:59.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:59 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:59.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:10:59 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:59.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:59 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:59.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:59 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:10:59.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:10:59 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:00.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:00 smithi174 ceph-mon[21825]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:00.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:00 smithi174 ceph-mon[26730]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:00.411 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:00 smithi192 ceph-mon[22835]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:02.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:02 smithi192 ceph-mon[22835]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:02.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:02 smithi174 ceph-mon[21825]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:02.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:02 smithi174 ceph-mon[26730]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:02.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:11:02 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:11:02] "GET /metrics HTTP/1.1" 200 215057 "" "Prometheus/2.33.4" 2024-04-22T22:11:02.770 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:11:02 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:11:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:11:04.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:04 smithi174 ceph-mon[26730]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:04.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:04 smithi174 ceph-mon[21825]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:04.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:04 smithi192 ceph-mon[22835]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:05.814 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:11:05 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:11:05] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:11:06.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:06 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:06.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:06 smithi174 ceph-mon[26730]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:06.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:06 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:06.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:06 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:06.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:06 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:06.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:06 smithi174 ceph-mon[21825]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:06.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:06 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:06.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:06 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:06.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:06 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:06.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:06 smithi192 ceph-mon[22835]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:06.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:06 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:06.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:06 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:08.417 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:08 smithi174 ceph-mon[21825]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:08.418 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:08 smithi174 ceph-mon[26730]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:08.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:08 smithi192 ceph-mon[22835]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:10.405 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:10 smithi174 ceph-mon[21825]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:10.406 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:10 smithi174 ceph-mon[26730]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:10.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:10 smithi192 ceph-mon[22835]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:12.400 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:12 smithi192 ceph-mon[22835]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:12.401 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:11:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:11:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:11:12.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:12 smithi174 ceph-mon[21825]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:12.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:12 smithi174 ceph-mon[26730]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:12.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:11:12 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:11:12] "GET /metrics HTTP/1.1" 200 215057 "" "Prometheus/2.33.4" 2024-04-22T22:11:13.313 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:11:13.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:11:13.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:11:13.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:11:13.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:11:13.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:11:14.563 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:14 smithi174 ceph-mon[21825]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:14.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:14 smithi174 ceph-mon[26730]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:14.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:14 smithi192 ceph-mon[22835]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:16.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:16 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:16.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:16 smithi174 ceph-mon[21825]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:16.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:16 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:16.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:16 smithi174 ceph-mon[26730]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:16.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:16 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:16.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:16 smithi192 ceph-mon[22835]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:18.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:18 smithi174 ceph-mon[21825]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:18.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:18 smithi174 ceph-mon[26730]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:18.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:11:18 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:11:18] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:11:18.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:18 smithi192 ceph-mon[22835]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:19.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:19 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:19.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:19 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:19.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:19 smithi192 ceph-mon[22835]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:19.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:19 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:19.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:19 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:19.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:19 smithi174 ceph-mon[21825]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:19.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:19 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:19.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:19 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:19.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:19 smithi174 ceph-mon[26730]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:22.313 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:22 smithi192 ceph-mon[22835]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:22.563 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:22 smithi174 ceph-mon[21825]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:22.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:22 smithi174 ceph-mon[26730]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:22.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:11:22 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:11:22] "GET /metrics HTTP/1.1" 200 215010 "" "Prometheus/2.33.4" 2024-04-22T22:11:22.805 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:11:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:11:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:11:24.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:24 smithi192 ceph-mon[22835]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:24.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:24 smithi174 ceph-mon[21825]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:24.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:24 smithi174 ceph-mon[26730]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:25.814 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:11:25 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:11:25] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:11:26.563 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:26 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:26.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:26 smithi174 ceph-mon[21825]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:26.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:26 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:26.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:26 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:26.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:26 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:26.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:26 smithi174 ceph-mon[26730]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:26.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:26 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:26.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:26 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:26.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:26 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:26.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:26 smithi192 ceph-mon[22835]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:26.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:26 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:26.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:26 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:28.424 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:28 smithi174 ceph-mon[21825]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:28.424 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:28 smithi174 ceph-mon[26730]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:28.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:28 smithi192 ceph-mon[22835]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:30.475 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:30 smithi174 ceph-mon[21825]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:30.476 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:30 smithi174 ceph-mon[26730]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:30.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:30 smithi192 ceph-mon[22835]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:32.386 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:32 smithi192 ceph-mon[22835]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:32.563 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:32 smithi174 ceph-mon[21825]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:32.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:32 smithi174 ceph-mon[26730]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:32.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:11:32 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:11:32] "GET /metrics HTTP/1.1" 200 215010 "" "Prometheus/2.33.4" 2024-04-22T22:11:32.813 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:11:32 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:11:32] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:11:34.563 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:34 smithi174 ceph-mon[21825]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:34.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:34 smithi174 ceph-mon[26730]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:34.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:34 smithi192 ceph-mon[22835]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:36.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:36 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:36.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:36 smithi192 ceph-mon[22835]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:36.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:36 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:36.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:36 smithi174 ceph-mon[26730]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:36.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:36 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:36.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:36 smithi174 ceph-mon[21825]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:38.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:38 smithi174 ceph-mon[21825]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:38.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:38 smithi174 ceph-mon[26730]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:38.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:11:38 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:11:38] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:11:38.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:38 smithi192 ceph-mon[22835]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:39.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:39 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:39.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:39 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:39.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:39 smithi192 ceph-mon[22835]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:39.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:39 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:39.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:39 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:39.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:39 smithi174 ceph-mon[21825]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:39.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:39 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:39.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:39 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:39.565 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:39 smithi174 ceph-mon[26730]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:42.313 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:42 smithi192 ceph-mon[22835]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:42.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:42 smithi174 ceph-mon[21825]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:42.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:42 smithi174 ceph-mon[26730]: pgmap v208: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:42.564 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:11:42 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:11:42] "GET /metrics HTTP/1.1" 200 215010 "" "Prometheus/2.33.4" 2024-04-22T22:11:42.722 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:11:42 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:11:42] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:11:44.505 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:44 smithi192 ceph-mon[22835]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:44.563 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:44 smithi174 ceph-mon[21825]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:44.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:44 smithi174 ceph-mon[26730]: pgmap v209: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:45.313 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:11:45.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:45 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:11:45.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:11:45.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:45 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:11:45.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-22T22:11:45.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:45 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-22T22:11:45.813 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:11:45 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:11:45] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:11:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:46 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:46 smithi192 ceph-mon[22835]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:46 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:46 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:46 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:46.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:46 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:46.563 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:46 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:46.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:46 smithi174 ceph-mon[26730]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:46.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:46 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:46.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:46 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:46.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:46 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:46.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:46 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:46.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:46 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:46.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:46 smithi174 ceph-mon[21825]: pgmap v210: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:46.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:46 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:46.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:46 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:46.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:46 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:46.565 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:46 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:48.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:48 smithi174 ceph-mon[21825]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:48.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:48 smithi174 ceph-mon[26730]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:48.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:48 smithi192 ceph-mon[22835]: pgmap v211: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:50.375 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:50 smithi174 ceph-mon[21825]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:50.376 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:50 smithi174 ceph-mon[26730]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:50.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:50 smithi192 ceph-mon[22835]: pgmap v212: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:52.366 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:52 smithi192 ceph-mon[22835]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:52.465 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:52 smithi174 ceph-mon[21825]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:52.465 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:52 smithi174 ceph-mon[26730]: pgmap v213: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:52.466 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:11:52 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:11:52] "GET /metrics HTTP/1.1" 200 215009 "" "Prometheus/2.33.4" 2024-04-22T22:11:52.813 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:11:52 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:11:52] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:11:54.564 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:54 smithi174 ceph-mon[21825]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:54.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:54 smithi174 ceph-mon[26730]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:54.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:54 smithi192 ceph-mon[22835]: pgmap v214: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:56.313 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:56 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:56.314 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:56 smithi192 ceph-mon[22835]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:56.563 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:56 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:56.563 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:56 smithi174 ceph-mon[21825]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:56.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:56 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:11:56.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:56 smithi174 ceph-mon[26730]: pgmap v215: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:11:58.420 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:58 smithi174 ceph-mon[21825]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:58.420 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:58 smithi174 ceph-mon[26730]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:58.421 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:58 smithi192 ceph-mon[22835]: pgmap v216: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:58.813 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:11:58 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:11:58] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:11:59.813 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:59 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:59.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:59 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:59.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:11:59 smithi174 ceph-mon[21825]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:59.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:59 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:59.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:59 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:59.814 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:11:59 smithi174 ceph-mon[26730]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:11:59.814 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:59 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:59.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:59 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:11:59.815 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:11:59 smithi192 ceph-mon[22835]: pgmap v217: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:00.797 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch ps' 2024-04-22T22:12:02.344 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:02 smithi174 ceph-mon[26730]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:12:02.344 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:02 smithi174 ceph-mon[21825]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:12:02.344 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:12:02 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:12:02] "GET /metrics HTTP/1.1" 200 215011 "" "Prometheus/2.33.4" 2024-04-22T22:12:02.384 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:02 smithi192 ceph-mon[22835]: pgmap v218: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:12:02.639 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:12:02 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:12:02] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:12:02.954 INFO:teuthology.orchestra.run.smithi174.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-22T22:12:02.954 INFO:teuthology.orchestra.run.smithi174.stdout:agent.smithi174 smithi174 running 17s ago 10m - - 2024-04-22T22:12:02.954 INFO:teuthology.orchestra.run.smithi174.stdout:agent.smithi192 smithi192 running 4s ago 9m - - 2024-04-22T22:12:02.954 INFO:teuthology.orchestra.run.smithi174.stdout:alertmanager.a smithi174 *:9093,9094 running (4m) 17s ago 4m 25.7M - ba2b418f427c 1e9de7a331d8 2024-04-22T22:12:02.954 INFO:teuthology.orchestra.run.smithi174.stdout:grafana.a smithi192 *:3000 running (4m) 4s ago 4m 57.0M - 8.3.5 dad864ee21e9 4787c8784d2e 2024-04-22T22:12:02.954 INFO:teuthology.orchestra.run.smithi174.stdout:iscsi.foo.smithi174.upousg smithi174 running (3m) 17s ago 3m 48.7M - 3.5 e1d6a67b021e cc36500f1ff2 2024-04-22T22:12:02.954 INFO:teuthology.orchestra.run.smithi174.stdout:mgr.x smithi192 *:8443 running (9m) 4s ago 9m 412M - 17.2.0 e1d6a67b021e bd398d520007 2024-04-22T22:12:02.954 INFO:teuthology.orchestra.run.smithi174.stdout:mgr.y smithi174 *:9283 running (10m) 17s ago 10m 465M - 17.2.0 e1d6a67b021e 739b59b845dc 2024-04-22T22:12:02.954 INFO:teuthology.orchestra.run.smithi174.stdout:mon.a smithi174 running (10m) 17s ago 10m 54.0M 2048M 17.2.0 e1d6a67b021e d85a5efa7dc5 2024-04-22T22:12:02.954 INFO:teuthology.orchestra.run.smithi174.stdout:mon.b smithi192 running (9m) 4s ago 9m 41.3M 2048M 17.2.0 e1d6a67b021e 4d13e5913d29 2024-04-22T22:12:02.954 INFO:teuthology.orchestra.run.smithi174.stdout:mon.c smithi174 running (9m) 17s ago 9m 39.8M 2048M 17.2.0 e1d6a67b021e 260b4acfe3f8 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:node-exporter.a smithi174 *:9100 running (5m) 17s ago 5m 16.2M - 1dbe0e931976 5e1c68495a2c 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:node-exporter.b smithi192 *:9100 running (5m) 4s ago 5m 18.1M - 1dbe0e931976 71966acd94b1 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:osd.0 smithi174 running (8m) 17s ago 8m 50.7M 4096M 17.2.0 e1d6a67b021e b5aa9f2bad5d 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:osd.1 smithi174 running (8m) 17s ago 8m 54.5M 4096M 17.2.0 e1d6a67b021e 926b9f070a23 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:osd.2 smithi174 running (8m) 17s ago 8m 46.0M 4096M 17.2.0 e1d6a67b021e 830775ff8596 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:osd.3 smithi174 running (7m) 17s ago 7m 49.7M 4096M 17.2.0 e1d6a67b021e 0922e1ebb4ed 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:osd.4 smithi192 running (7m) 4s ago 7m 52.2M 4096M 17.2.0 e1d6a67b021e c77c665e8b26 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:osd.5 smithi192 running (6m) 4s ago 6m 47.4M 4096M 17.2.0 e1d6a67b021e 67848ad026e0 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:osd.6 smithi192 running (6m) 4s ago 6m 46.5M 4096M 17.2.0 e1d6a67b021e d75a88722468 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:osd.7 smithi192 running (6m) 4s ago 6m 52.3M 4096M 17.2.0 e1d6a67b021e a4170e31930d 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:prometheus.a smithi192 *:9095 running (4m) 4s ago 5m 63.6M - 514e6a882f6e dfa924a46ee2 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:rgw.foo.smithi174.eqkeoc smithi174 *:8000 running (3m) 17s ago 3m 69.9M - 17.2.0 e1d6a67b021e 82a13f9621e8 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:rgw.foo.smithi192.ihnxdd smithi192 *:8000 running (3m) 4s ago 3m 71.5M - 17.2.0 e1d6a67b021e 1ad669d97de3 2024-04-22T22:12:02.955 INFO:teuthology.orchestra.run.smithi174.stdout:rgw.smpl.smithi174.ggrvao smithi174 *:80 running (3m) 17s ago 3m 69.7M - 17.2.0 e1d6a67b021e 5b3499dd9c27 2024-04-22T22:12:02.956 INFO:teuthology.orchestra.run.smithi174.stdout:rgw.smpl.smithi192.nzyxdu smithi192 *:80 running (3m) 4s ago 3m 70.9M - 17.2.0 e1d6a67b021e bcc184444db6 2024-04-22T22:12:03.614 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions' 2024-04-22T22:12:04.169 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:04 smithi174 ceph-mon[21825]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:12:04.169 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:04 smithi174 ceph-mon[21825]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:04.169 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:04 smithi174 ceph-mon[26730]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:12:04.169 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:04 smithi174 ceph-mon[26730]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:04.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:04 smithi192 ceph-mon[22835]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:12:04.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:04 smithi192 ceph-mon[22835]: pgmap v219: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:05.715 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-22T22:12:05.715 INFO:teuthology.orchestra.run.smithi174.stdout: "mon": { 2024-04-22T22:12:05.715 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-22T22:12:05.715 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:12:05.715 INFO:teuthology.orchestra.run.smithi174.stdout: "mgr": { 2024-04-22T22:12:05.716 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-22T22:12:05.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:12:05.716 INFO:teuthology.orchestra.run.smithi174.stdout: "osd": { 2024-04-22T22:12:05.716 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-22T22:12:05.716 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:12:05.716 INFO:teuthology.orchestra.run.smithi174.stdout: "mds": {}, 2024-04-22T22:12:05.716 INFO:teuthology.orchestra.run.smithi174.stdout: "rgw": { 2024-04-22T22:12:05.717 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-22T22:12:05.717 INFO:teuthology.orchestra.run.smithi174.stdout: }, 2024-04-22T22:12:05.717 INFO:teuthology.orchestra.run.smithi174.stdout: "overall": { 2024-04-22T22:12:05.717 INFO:teuthology.orchestra.run.smithi174.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-22T22:12:05.717 INFO:teuthology.orchestra.run.smithi174.stdout: } 2024-04-22T22:12:05.717 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-22T22:12:06.017 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:12:05 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.174 - - [22/Apr/2024:22:12:05] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:12:06.313 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:06 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:12:06.313 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:06 smithi174 ceph-mon[21825]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:12:06.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:06 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:12:06.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:06 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:12:06.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:06 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/226554165' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:12:06.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:06 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:12:06.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:06 smithi174 ceph-mon[26730]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:12:06.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:06 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:12:06.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:06 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:12:06.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:06 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/226554165' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:12:06.452 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph orch upgrade status' 2024-04-22T22:12:06.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:06 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:12:06.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:06 smithi192 ceph-mon[22835]: pgmap v220: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:12:06.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:06 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:12:06.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:06 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:12:06.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:06 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/226554165' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:12:08.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:08 smithi174 ceph-mon[26730]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:08.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:08 smithi174 ceph-mon[21825]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:08.469 INFO:teuthology.orchestra.run.smithi174.stdout:{ 2024-04-22T22:12:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f", 2024-04-22T22:12:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: "in_progress": true, 2024-04-22T22:12:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: "services_complete": [], 2024-04-22T22:12:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: "progress": "", 2024-04-22T22:12:08.470 INFO:teuthology.orchestra.run.smithi174.stdout: "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-22T22:12:08.470 INFO:teuthology.orchestra.run.smithi174.stdout:} 2024-04-22T22:12:08.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:08 smithi192 ceph-mon[22835]: pgmap v221: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:09.175 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph health detail' 2024-04-22T22:12:10.378 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:10 smithi174 ceph-mon[21825]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:12:10.378 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:10 smithi174 ceph-mon[21825]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:10.379 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:10 smithi174 ceph-mon[26730]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:12:10.379 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:10 smithi174 ceph-mon[26730]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:10.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:10 smithi192 ceph-mon[22835]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-22T22:12:10.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:10 smithi192 ceph-mon[22835]: pgmap v222: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:12.102 INFO:teuthology.orchestra.run.smithi174.stdout:HEALTH_WARN Upgrade: failed to pull target image 2024-04-22T22:12:12.102 INFO:teuthology.orchestra.run.smithi174.stdout:[WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-22T22:12:12.102 INFO:teuthology.orchestra.run.smithi174.stdout: host smithi174 `cephadm pull` failed: cephadm exited with an error code: 1, stderr: Pulling container image quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f... 2024-04-22T22:12:12.102 INFO:teuthology.orchestra.run.smithi174.stdout:Non-zero exit code 125 from /bin/podman pull quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:12:12.102 INFO:teuthology.orchestra.run.smithi174.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f... 2024-04-22T22:12:12.103 INFO:teuthology.orchestra.run.smithi174.stdout:/bin/podman: stderr Error: initializing source docker://quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f: reading manifest 430e09df97c8fc7dc2b2ae424f68ed11366c540f in quay.ceph.io/ceph-ci/ceph: manifest unknown 2024-04-22T22:12:12.103 INFO:teuthology.orchestra.run.smithi174.stdout:ERROR: Failed command: /bin/podman pull quay.ceph.io/ceph-ci/ceph:430e09df97c8fc7dc2b2ae424f68ed11366c540f 2024-04-22T22:12:12.197 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:12 smithi174 ceph-mon[21825]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:12:12.197 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:12 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/2911145815' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T22:12:12.197 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:12 smithi174 ceph-mon[26730]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:12:12.197 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:12 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/2911145815' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T22:12:12.385 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:12 smithi192 ceph-mon[22835]: pgmap v223: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:12:12.385 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:12 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/2911145815' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-22T22:12:12.563 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:12:12 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:12:12] "GET /metrics HTTP/1.1" 200 215011 "" "Prometheus/2.33.4" 2024-04-22T22:12:12.813 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:12:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:12:12] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:12:12.891 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-22T22:12:13.563 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:13 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:12:13.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:13 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:12:13.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:13 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-22T22:12:14.462 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:14 smithi174 ceph-mon[21825]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:14.462 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:14 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:12:14.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:14 smithi174 ceph-mon[26730]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:14.462 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:14 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:12:14.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:14 smithi192 ceph-mon[22835]: pgmap v224: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:14.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:14 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-22T22:12:15.569 INFO:teuthology.orchestra.run.smithi174.stdout:true 2024-04-22T22:12:16.159 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-22T22:12:16.313 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:16 smithi174 ceph-mon[21825]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:12:16.313 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:16 smithi174 ceph-mon[21825]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:12:16.313 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:16 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/740300738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:12:16.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:16 smithi174 ceph-mon[26730]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:12:16.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:16 smithi174 ceph-mon[26730]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:12:16.314 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:16 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/740300738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:12:16.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:16 smithi192 ceph-mon[22835]: from='client.14949 -' entity='client.iscsi.foo.smithi174.upousg' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-22T22:12:16.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:16 smithi192 ceph-mon[22835]: pgmap v225: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-22T22:12:16.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:16 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/740300738' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:12:18.521 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:18 smithi174 ceph-mon[21825]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:18.521 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:18 smithi192 ceph-mon[22835]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:18.522 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:18 smithi174 ceph-mon[26730]: pgmap v226: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-22T22:12:18.813 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:12:18 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 172.21.15.192 - - [22/Apr/2024:22:12:18] "POST /data HTTP/1.1" 200 46 "" "Python-urllib/3.9" 2024-04-22T22:12:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:12:18.967 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 1469, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi174 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-22T22:12:19.198 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b71aab70a00a442480511c1049f2cf9d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 1469, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_c66b8bf2efd3f3988ac1851474c2f98eb2ca30d9/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi174 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-22T22:12:19.201 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-22T22:12:19.212 INFO:tasks.cephadm:Teardown begin 2024-04-22T22:12:19.213 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:12:19.243 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:19 smithi174 ceph-mon[21825]: from='client.? 172.21.15.174:0/149833476' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:12:19.243 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:19 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:12:19.244 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:19 smithi174 ceph-mon[21825]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:12:19.244 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:19 smithi174 ceph-mon[26730]: from='client.? 172.21.15.174:0/149833476' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:12:19.244 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:19 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:12:19.244 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:19 smithi174 ceph-mon[26730]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:12:19.248 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:12:19.282 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-22T22:12:19.282 DEBUG:teuthology.orchestra.run.smithi174:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T22:12:19.304 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-22T22:12:19.339 INFO:tasks.cephadm:Stopping all daemons... 2024-04-22T22:12:19.340 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-22T22:12:19.340 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mon.a 2024-04-22T22:12:19.538 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:19 smithi174 systemd[1]: Stopping Ceph mon.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:12:19.538 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:19 smithi174 bash[59190]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon.a" found: no such container 2024-04-22T22:12:19.563 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:19 smithi192 ceph-mon[22835]: from='client.? 172.21.15.174:0/149833476' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-22T22:12:19.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:19 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:12:19.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:19 smithi192 ceph-mon[22835]: from='mgr.24349 172.21.15.174:0/514812652' entity='mgr.y' 2024-04-22T22:12:19.813 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:19 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a[21821]: 2024-04-22T22:12:19.563+0000 7f8f38986700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:12:19.813 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:19 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a[21821]: 2024-04-22T22:12:19.563+0000 7f8f38986700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-22T22:12:19.814 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:19 smithi174 podman[59200]: 2024-04-22 22:12:19.767630312 +0000 UTC m=+0.324584753 container died d85a5efa7dc5763848bb269686567df898f06fd442d4ff43a9b6d16923597215 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a, release=754, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_CLEAN=True, RELEASE=HEAD, distribution-scope=public, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux ) 2024-04-22T22:12:20.313 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:20 smithi174 podman[59200]: 2024-04-22 22:12:20.255169103 +0000 UTC m=+0.812123593 container cleanup d85a5efa7dc5763848bb269686567df898f06fd442d4ff43a9b6d16923597215 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.k8s.display-name=CentOS Stream 8, name=centos-stream, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_BRANCH=HEAD, RELEASE=HEAD, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True) 2024-04-22T22:12:20.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:20 smithi174 bash[59200]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-a 2024-04-22T22:12:20.314 INFO:journalctl@ceph.mon.a.smithi174.stdout:Apr 22 22:12:20 smithi174 bash[59228]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon.a" found: no such container 2024-04-22T22:12:20.478 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mon.a.service' 2024-04-22T22:12:20.529 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:12:20.529 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-22T22:12:20.529 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-22T22:12:20.529 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mon.c 2024-04-22T22:12:20.778 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:20 smithi174 systemd[1]: Stopping Ceph mon.c for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:12:20.778 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:20 smithi174 bash[59281]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon.c" found: no such container 2024-04-22T22:12:20.779 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:20 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-c[26713]: 2024-04-22T22:12:20.742+0000 7ff663f30700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:12:20.779 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:20 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-c[26713]: 2024-04-22T22:12:20.742+0000 7ff663f30700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-22T22:12:21.063 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:20 smithi174 podman[59293]: 2024-04-22 22:12:20.804144905 +0000 UTC m=+0.173927471 container died 260b4acfe3f883206d0cf7c6ef164c45b366648e664b1a8af0a77b7795514fe3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-c, architecture=x86_64, GIT_CLEAN=True, vendor=Red Hat, Inc., version=8, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, ceph=True, release=754, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, maintainer=Guillaume Abrioux , vcs-type=git, io.buildah.version=1.19.8, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T22:12:21.563 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:21 smithi174 podman[59293]: 2024-04-22 22:12:21.351574829 +0000 UTC m=+0.721357440 container cleanup 260b4acfe3f883206d0cf7c6ef164c45b366648e664b1a8af0a77b7795514fe3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-c, release=754, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, vcs-type=git, distribution-scope=public, architecture=x86_64, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, ceph=True) 2024-04-22T22:12:21.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:21 smithi174 bash[59293]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-c 2024-04-22T22:12:21.564 INFO:journalctl@ceph.mon.c.smithi174.stdout:Apr 22 22:12:21 smithi174 bash[59319]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon.c" found: no such container 2024-04-22T22:12:21.624 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mon.c.service' 2024-04-22T22:12:21.662 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:12:21.662 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-22T22:12:21.663 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-22T22:12:21.663 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mon.b 2024-04-22T22:12:22.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:21 smithi192 systemd[1]: Stopping Ceph mon.b for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:12:22.064 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:21 smithi192 bash[48409]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon.b" found: no such container 2024-04-22T22:12:22.563 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:12:22 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:12:22] "GET /metrics HTTP/1.1" 200 215017 "" "Prometheus/2.33.4" 2024-04-22T22:12:22.564 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:12:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x[24373]: ::ffff:172.21.15.192 - - [22/Apr/2024:22:12:22] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-22T22:12:22.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-b[22818]: 2024-04-22T22:12:22.187+0000 7f6edadb9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:12:22.564 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-b[22818]: 2024-04-22T22:12:22.187+0000 7f6edadb9700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-22T22:12:22.565 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:22 smithi192 podman[48422]: 2024-04-22 22:12:22.413590818 +0000 UTC m=+0.635467331 container died 4d13e5913d298c033de53e023d3a388b37528df95532fad612644363f81b365c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-b, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, name=centos-stream, vcs-type=git) 2024-04-22T22:12:22.834 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:22 smithi192 podman[48422]: 2024-04-22 22:12:22.838091967 +0000 UTC m=+1.059968473 container cleanup 4d13e5913d298c033de53e023d3a388b37528df95532fad612644363f81b365c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-b, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, version=8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, name=centos-stream, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public) 2024-04-22T22:12:22.834 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:22 smithi192 bash[48422]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon-b 2024-04-22T22:12:23.139 INFO:journalctl@ceph.mon.b.smithi192.stdout:Apr 22 22:12:22 smithi192 bash[48449]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mon.b" found: no such container 2024-04-22T22:12:23.144 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mon.b.service' 2024-04-22T22:12:23.189 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:12:23.189 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-22T22:12:23.190 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-22T22:12:23.190 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mgr.y 2024-04-22T22:12:23.313 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:12:23 smithi174 systemd[1]: Stopping Ceph mgr.y for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:12:23.314 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:12:23 smithi174 bash[59462]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr.y" found: no such container 2024-04-22T22:12:23.714 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:12:23 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:12:23.422+0000 7f9f568e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:12:23.715 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:12:23 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y[22059]: 2024-04-22T22:12:23.422+0000 7f9f568e1700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-22T22:12:23.715 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:12:23 smithi174 podman[59484]: 2024-04-22 22:12:23.453594014 +0000 UTC m=+0.157109297 container died 739b59b845dcb92dad8da069947f1db01e078d686ccc4e38685d4ac5a462a7b0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, distribution-scope=public, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , architecture=x86_64, release=754, io.openshift.tags=base centos centos-stream, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, version=8) 2024-04-22T22:12:24.063 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:12:23 smithi174 podman[59484]: 2024-04-22 22:12:23.915035524 +0000 UTC m=+0.618550809 container cleanup 739b59b845dcb92dad8da069947f1db01e078d686ccc4e38685d4ac5a462a7b0 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, RELEASE=HEAD, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.openshift.expose-services=, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-22T22:12:24.063 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:12:23 smithi174 bash[59484]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-y 2024-04-22T22:12:24.064 INFO:journalctl@ceph.mgr.y.smithi174.stdout:Apr 22 22:12:23 smithi174 bash[59546]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr.y" found: no such container 2024-04-22T22:12:24.187 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-22T22:12:24.227 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:12:24.227 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-22T22:12:24.227 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-22T22:12:24.227 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mgr.x 2024-04-22T22:12:24.529 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:12:24 smithi192 systemd[1]: Stopping Ceph mgr.x for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:12:24.814 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:12:24 smithi192 bash[48550]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr.x" found: no such container 2024-04-22T22:12:24.814 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:12:24 smithi192 podman[48567]: 2024-04-22 22:12:24.779180703 +0000 UTC m=+0.118981510 container died bd398d520007a4d3c35cfce4ea5a36a274057d8ef0a7c5c437c407d9723a65f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, vendor=Red Hat, Inc., GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_BRANCH=HEAD, version=8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git) 2024-04-22T22:12:25.291 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:12:25 smithi192 podman[48567]: 2024-04-22 22:12:25.099754473 +0000 UTC m=+0.439555282 container cleanup bd398d520007a4d3c35cfce4ea5a36a274057d8ef0a7c5c437c407d9723a65f1 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, com.redhat.component=centos-stream-container, name=centos-stream, RELEASE=HEAD, io.openshift.expose-services=) 2024-04-22T22:12:25.291 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:12:25 smithi192 bash[48567]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr-x 2024-04-22T22:12:25.291 INFO:journalctl@ceph.mgr.x.smithi192.stdout:Apr 22 22:12:25 smithi192 bash[48594]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-mgr.x" found: no such container 2024-04-22T22:12:25.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-22T22:12:25.332 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:12:25.332 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-22T22:12:25.333 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-22T22:12:25.333 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.0 2024-04-22T22:12:25.563 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:25 smithi174 systemd[1]: Stopping Ceph osd.0 for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:12:26.046 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:25 smithi174 bash[59761]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-22T22:12:26.046 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:25 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0[29984]: 2024-04-22T22:12:25.738+0000 7f3b10b37700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:12:26.046 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:25 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0[29984]: 2024-04-22T22:12:25.738+0000 7f3b10b37700 -1 osd.0 93 *** Got signal Terminated *** 2024-04-22T22:12:26.047 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:25 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0[29984]: 2024-04-22T22:12:25.738+0000 7f3b10b37700 -1 osd.0 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:12:31.313 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:31 smithi174 podman[59792]: 2024-04-22 22:12:31.282295021 +0000 UTC m=+5.670492990 container died b5aa9f2bad5d17b7627692ca88f6c26a7a634a03a64e516e37d0890f2dc810e9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0, maintainer=Guillaume Abrioux , vcs-type=git, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, architecture=x86_64, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, GIT_BRANCH=HEAD, version=8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0) 2024-04-22T22:12:31.813 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:31 smithi174 podman[59792]: 2024-04-22 22:12:31.694593285 +0000 UTC m=+6.082791263 container cleanup b5aa9f2bad5d17b7627692ca88f6c26a7a634a03a64e516e37d0890f2dc810e9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, release=754, version=8, GIT_CLEAN=True, vcs-type=git, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, GIT_BRANCH=HEAD) 2024-04-22T22:12:31.814 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:31 smithi174 bash[59792]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0 2024-04-22T22:12:32.314 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:31 smithi174 podman[60156]: 2024-04-22 22:12:31.887747941 +0000 UTC m=+0.637744077 container remove b5aa9f2bad5d17b7627692ca88f6c26a7a634a03a64e516e37d0890f2dc810e9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0, architecture=x86_64, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, ceph=True, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, vcs-type=git, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.openshift.expose-services=) 2024-04-22T22:12:32.314 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:31 smithi174 bash[60170]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.0" found: no such container 2024-04-22T22:12:33.421 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:33 smithi174 podman[60261]: 2024-04-22 22:12:33.092361636 +0000 UTC m=+0.022916339 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:12:33.421 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:33 smithi174 podman[60261]: 2024-04-22 22:12:33.236312715 +0000 UTC m=+0.166867454 container create f2489a31de70b68455cd968d3d10c9ef5393e73c0adc4dca0d5dbb97577c08aa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0-deactivate, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_BRANCH=HEAD, version=8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, io.buildah.version=1.19.8, ceph=True, vendor=Red Hat, Inc., GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux ) 2024-04-22T22:12:33.421 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:33 smithi174 podman[60261]: 2024-04-22 22:12:33.446728708 +0000 UTC m=+0.377283407 container init f2489a31de70b68455cd968d3d10c9ef5393e73c0adc4dca0d5dbb97577c08aa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0-deactivate, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, version=8, release=754, maintainer=Guillaume Abrioux , distribution-scope=public, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, ceph=True) 2024-04-22T22:12:33.692 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:33 smithi174 podman[60261]: 2024-04-22 22:12:33.450100196 +0000 UTC m=+0.380654898 container start f2489a31de70b68455cd968d3d10c9ef5393e73c0adc4dca0d5dbb97577c08aa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, version=8, vcs-type=git, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, release=754, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, ceph=True, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-22T22:12:33.693 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:33 smithi174 podman[60261]: 2024-04-22 22:12:33.563661202 +0000 UTC m=+0.494215914 container attach f2489a31de70b68455cd968d3d10c9ef5393e73c0adc4dca0d5dbb97577c08aa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0-deactivate, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, release=754, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, distribution-scope=public, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, RELEASE=HEAD, com.redhat.component=centos-stream-container) 2024-04-22T22:12:34.225 INFO:journalctl@ceph.osd.0.smithi174.stdout:Apr 22 22:12:33 smithi174 podman[60261]: 2024-04-22 22:12:33.970659806 +0000 UTC m=+0.901214504 container died f2489a31de70b68455cd968d3d10c9ef5393e73c0adc4dca0d5dbb97577c08aa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, version=8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, RELEASE=HEAD, ceph=True, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git) 2024-04-22T22:12:34.604 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.0.service' 2024-04-22T22:12:34.648 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:12:34.648 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-22T22:12:34.648 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-22T22:12:34.649 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.1 2024-04-22T22:12:35.054 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:34 smithi174 systemd[1]: Stopping Ceph osd.1 for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:12:35.313 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:35 smithi174 bash[60390]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-22T22:12:35.314 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1[32874]: 2024-04-22T22:12:35.223+0000 7fc8c63a0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:12:35.314 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1[32874]: 2024-04-22T22:12:35.223+0000 7fc8c63a0700 -1 osd.1 93 *** Got signal Terminated *** 2024-04-22T22:12:35.314 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:35 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1[32874]: 2024-04-22T22:12:35.223+0000 7fc8c63a0700 -1 osd.1 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:12:40.957 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:40 smithi174 podman[60404]: 2024-04-22 22:12:40.694144641 +0000 UTC m=+5.571790804 container died 926b9f070a23561bf641f222a81a4c7ab4bf433e0983477fb7e248f30a642b56 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, distribution-scope=public, vendor=Red Hat, Inc., io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, com.redhat.component=centos-stream-container, architecture=x86_64, ceph=True, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, name=centos-stream, release=754, version=8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T22:12:41.207 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:41 smithi174 podman[60404]: 2024-04-22 22:12:41.190623529 +0000 UTC m=+6.068269682 container cleanup 926b9f070a23561bf641f222a81a4c7ab4bf433e0983477fb7e248f30a642b56 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, architecture=x86_64, release=754, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, version=8, ceph=True, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream) 2024-04-22T22:12:41.207 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:41 smithi174 bash[60404]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1 2024-04-22T22:12:41.207 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:41 smithi174 bash[60435]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.1" found: no such container 2024-04-22T22:12:41.746 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:41 smithi174 podman[60422]: 2024-04-22 22:12:41.425282116 +0000 UTC m=+0.726621546 container remove 926b9f070a23561bf641f222a81a4c7ab4bf433e0983477fb7e248f30a642b56 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, release=754, name=centos-stream, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8) 2024-04-22T22:12:41.747 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:41 smithi174 podman[60502]: 2024-04-22 22:12:41.535885614 +0000 UTC m=+0.018980880 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:12:41.747 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:41 smithi174 podman[60502]: 2024-04-22 22:12:41.651300888 +0000 UTC m=+0.134396136 container create 0ebb9af7637f90bcfb1463954d82ca130929f3e96632b45e0bdc06b0351dec66 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1-deactivate, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, vcs-type=git, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-22T22:12:42.063 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:41 smithi174 podman[60502]: 2024-04-22 22:12:41.89713907 +0000 UTC m=+0.380234317 container init 0ebb9af7637f90bcfb1463954d82ca130929f3e96632b45e0bdc06b0351dec66 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, RELEASE=HEAD, name=centos-stream, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , version=8, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, distribution-scope=public, GIT_CLEAN=True, vcs-type=git, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.openshift.expose-services=) 2024-04-22T22:12:42.064 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:41 smithi174 podman[60502]: 2024-04-22 22:12:41.900461373 +0000 UTC m=+0.383556624 container start 0ebb9af7637f90bcfb1463954d82ca130929f3e96632b45e0bdc06b0351dec66 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.buildah.version=1.19.8, vcs-type=git, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD) 2024-04-22T22:12:42.064 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:41 smithi174 podman[60502]: 2024-04-22 22:12:41.980192142 +0000 UTC m=+0.463287390 container attach 0ebb9af7637f90bcfb1463954d82ca130929f3e96632b45e0bdc06b0351dec66 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, vcs-type=git, version=8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, architecture=x86_64, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T22:12:42.506 INFO:journalctl@ceph.osd.1.smithi174.stdout:Apr 22 22:12:42 smithi174 podman[60502]: 2024-04-22 22:12:42.189577623 +0000 UTC m=+0.672672877 container died 0ebb9af7637f90bcfb1463954d82ca130929f3e96632b45e0bdc06b0351dec66 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, architecture=x86_64, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, GIT_CLEAN=True, name=centos-stream, vcs-type=git, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc.) 2024-04-22T22:12:42.886 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.1.service' 2024-04-22T22:12:42.927 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:12:42.928 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-22T22:12:42.928 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-22T22:12:42.928 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.2 2024-04-22T22:12:43.144 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:43 smithi174 systemd[1]: Stopping Ceph osd.2 for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:12:43.144 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:43 smithi174 bash[60580]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-22T22:12:43.563 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:43 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2[35799]: 2024-04-22T22:12:43.169+0000 7fd2f8699700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:12:43.564 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:43 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2[35799]: 2024-04-22T22:12:43.169+0000 7fd2f8699700 -1 osd.2 93 *** Got signal Terminated *** 2024-04-22T22:12:43.564 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:43 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2[35799]: 2024-04-22T22:12:43.169+0000 7fd2f8699700 -1 osd.2 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:12:48.762 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:48 smithi174 podman[60591]: 2024-04-22 22:12:48.523727033 +0000 UTC m=+5.455020586 container died 830775ff85960ed934cfe97e6f5faf645a41e6cd5dc65512c6ddc0c9646a47f6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , vcs-type=git, version=8, com.redhat.component=centos-stream-container, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, io.buildah.version=1.19.8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-22T22:12:49.063 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:48 smithi174 podman[60591]: 2024-04-22 22:12:48.998687642 +0000 UTC m=+5.929981192 container cleanup 830775ff85960ed934cfe97e6f5faf645a41e6cd5dc65512c6ddc0c9646a47f6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, name=centos-stream, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, distribution-scope=public) 2024-04-22T22:12:49.063 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:48 smithi174 bash[60591]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2 2024-04-22T22:12:49.064 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:49 smithi174 bash[60934]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.2" found: no such container 2024-04-22T22:12:49.683 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:49 smithi174 podman[60918]: 2024-04-22 22:12:49.372130981 +0000 UTC m=+0.843128904 container remove 830775ff85960ed934cfe97e6f5faf645a41e6cd5dc65512c6ddc0c9646a47f6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, version=8, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vcs-type=git, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, name=centos-stream, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-22T22:12:49.683 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:49 smithi174 podman[61043]: 2024-04-22 22:12:49.492145964 +0000 UTC m=+0.020773045 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:12:49.683 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:49 smithi174 podman[61043]: 2024-04-22 22:12:49.614720335 +0000 UTC m=+0.143347396 container create f3a990b0c7430815a653a894adb5221226007d6f6dbaefa2ca976cbd630f9dc5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2-deactivate, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , distribution-scope=public, name=centos-stream, GIT_BRANCH=HEAD, GIT_CLEAN=True, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, release=754, architecture=x86_64) 2024-04-22T22:12:50.063 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:49 smithi174 podman[61043]: 2024-04-22 22:12:49.808817586 +0000 UTC m=+0.337444650 container init f3a990b0c7430815a653a894adb5221226007d6f6dbaefa2ca976cbd630f9dc5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2-deactivate, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.openshift.expose-services=, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream) 2024-04-22T22:12:50.064 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:49 smithi174 podman[61043]: 2024-04-22 22:12:49.81244284 +0000 UTC m=+0.341069901 container start f3a990b0c7430815a653a894adb5221226007d6f6dbaefa2ca976cbd630f9dc5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , architecture=x86_64, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, release=754, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, ceph=True, distribution-scope=public) 2024-04-22T22:12:50.064 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:49 smithi174 podman[61043]: 2024-04-22 22:12:49.891820472 +0000 UTC m=+0.420447541 container attach f3a990b0c7430815a653a894adb5221226007d6f6dbaefa2ca976cbd630f9dc5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, io.openshift.expose-services=, maintainer=Guillaume Abrioux , ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, release=754, distribution-scope=public, architecture=x86_64, RELEASE=HEAD, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870) 2024-04-22T22:12:50.771 INFO:journalctl@ceph.osd.2.smithi174.stdout:Apr 22 22:12:50 smithi174 podman[61043]: 2024-04-22 22:12:50.535972087 +0000 UTC m=+1.064599148 container died f3a990b0c7430815a653a894adb5221226007d6f6dbaefa2ca976cbd630f9dc5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-2-deactivate, release=754, distribution-scope=public, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_CLEAN=True, architecture=x86_64, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream) 2024-04-22T22:12:51.154 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.2.service' 2024-04-22T22:12:51.196 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:12:51.196 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-22T22:12:51.196 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-22T22:12:51.196 DEBUG:teuthology.orchestra.run.smithi174:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.3 2024-04-22T22:12:51.429 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:51 smithi174 systemd[1]: Stopping Ceph osd.3 for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:12:51.813 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:51 smithi174 bash[61233]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-22T22:12:52.563 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:52 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3[38782]: 2024-04-22T22:12:52.397+0000 7f30be0a2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:12:52.563 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:52 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3[38782]: 2024-04-22T22:12:52.397+0000 7f30be0a2700 -1 osd.3 93 *** Got signal Terminated *** 2024-04-22T22:12:52.564 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:52 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3[38782]: 2024-04-22T22:12:52.397+0000 7f30be0a2700 -1 osd.3 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:12:53.564 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:12:53 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:12:53.197+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:12:54.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:12:53 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:12:53.604+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:12:54.425 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:54 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3[38782]: 2024-04-22T22:12:54.189+0000 7f30bb23e700 -1 osd.3 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.785323+0000 front 2024-04-22T22:12:29.785392+0000 (oldest deadline 2024-04-22T22:12:53.185908+0000) 2024-04-22T22:12:54.563 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:12:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:12:54.220+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:12:54.976 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:12:54 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:12:54.643+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:12:55.313 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:12:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:12:55.209+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:12:55.371 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:55 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3[38782]: 2024-04-22T22:12:55.140+0000 7f30bb23e700 -1 osd.3 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.785323+0000 front 2024-04-22T22:12:29.785392+0000 (oldest deadline 2024-04-22T22:12:53.185908+0000) 2024-04-22T22:12:55.639 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:12:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:12:55.410+0000 7f59e3860700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:30.511393+0000 front 2024-04-22T22:12:30.511452+0000 (oldest deadline 2024-04-22T22:12:55.211336+0000) 2024-04-22T22:12:56.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:12:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:12:55.676+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:12:56.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:12:55 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:12:55.664+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:12:56.563 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:56 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3[38782]: 2024-04-22T22:12:56.179+0000 7f30bb23e700 -1 osd.3 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.785323+0000 front 2024-04-22T22:12:29.785392+0000 (oldest deadline 2024-04-22T22:12:53.185908+0000) 2024-04-22T22:12:56.564 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:12:56 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:12:56.445+0000 7f59e3860700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:30.511393+0000 front 2024-04-22T22:12:30.511452+0000 (oldest deadline 2024-04-22T22:12:55.211336+0000) 2024-04-22T22:12:56.564 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:12:56 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:12:56.233+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:12:57.063 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:12:56 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:12:56.677+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:12:57.064 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:12:56 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:12:56.682+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:12:57.424 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:12:57 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:12:57.238+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:12:57.563 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:57 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3[38782]: 2024-04-22T22:12:57.220+0000 7f30bb23e700 -1 osd.3 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.785323+0000 front 2024-04-22T22:12:29.785392+0000 (oldest deadline 2024-04-22T22:12:53.185908+0000) 2024-04-22T22:12:57.674 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:12:57 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:12:57.450+0000 7f59e3860700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:30.511393+0000 front 2024-04-22T22:12:30.511452+0000 (oldest deadline 2024-04-22T22:12:55.211336+0000) 2024-04-22T22:12:58.006 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:57 smithi174 podman[61250]: 2024-04-22 22:12:57.770445828 +0000 UTC m=+6.306293242 container died 0922e1ebb4ed84970cdd3fa58be111b5dec43f8725b39046a402c36062d73486 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, vcs-type=git, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, name=centos-stream, io.openshift.expose-services=, distribution-scope=public, version=8, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD) 2024-04-22T22:12:58.063 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:12:57 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:12:57.700+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:12:58.064 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:12:57 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:12:57.717+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:12:58.313 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:58 smithi174 podman[61250]: 2024-04-22 22:12:58.18171027 +0000 UTC m=+6.717557670 container cleanup 0922e1ebb4ed84970cdd3fa58be111b5dec43f8725b39046a402c36062d73486 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, ceph=True, GIT_CLEAN=True, architecture=x86_64, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-22T22:12:58.313 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:58 smithi174 bash[61250]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3 2024-04-22T22:12:58.313 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:58 smithi174 bash[61595]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.3" found: no such container 2024-04-22T22:12:58.563 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:12:58 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:12:58.247+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:12:58.564 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:12:58 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:12:58.461+0000 7f59e3860700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:30.511393+0000 front 2024-04-22T22:12:30.511452+0000 (oldest deadline 2024-04-22T22:12:55.211336+0000) 2024-04-22T22:12:58.751 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:58 smithi174 podman[61580]: 2024-04-22 22:12:58.408322343 +0000 UTC m=+0.632520199 container remove 0922e1ebb4ed84970cdd3fa58be111b5dec43f8725b39046a402c36062d73486 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., release=754, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, ceph=True, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, vcs-type=git) 2024-04-22T22:12:58.751 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:58 smithi174 podman[61662]: 2024-04-22 22:12:58.524892976 +0000 UTC m=+0.018710929 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:12:58.751 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:58 smithi174 podman[61662]: 2024-04-22 22:12:58.625638423 +0000 UTC m=+0.119456361 container create 1be3c1e8c2f42b2021e30d9f593a6adb7b97999403c880a565fb320601a950ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, GIT_CLEAN=True, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, vcs-type=git, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T22:12:59.063 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:58 smithi174 podman[61662]: 2024-04-22 22:12:58.886056037 +0000 UTC m=+0.379873975 container init 1be3c1e8c2f42b2021e30d9f593a6adb7b97999403c880a565fb320601a950ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, GIT_CLEAN=True, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.component=centos-stream-container, architecture=x86_64, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, io.openshift.tags=base centos centos-stream) 2024-04-22T22:12:59.064 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:58 smithi174 podman[61662]: 2024-04-22 22:12:58.889431596 +0000 UTC m=+0.383249537 container start 1be3c1e8c2f42b2021e30d9f593a6adb7b97999403c880a565fb320601a950ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3-deactivate, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, release=754, ceph=True, RELEASE=HEAD, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, io.openshift.expose-services=, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-22T22:12:59.064 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:58 smithi174 podman[61662]: 2024-04-22 22:12:58.969386846 +0000 UTC m=+0.463204799 container attach 1be3c1e8c2f42b2021e30d9f593a6adb7b97999403c880a565fb320601a950ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, architecture=x86_64, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.buildah.version=1.19.8, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, release=754) 2024-04-22T22:12:59.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:12:58 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:12:58.749+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:12:59.064 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:12:58 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:12:58.716+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:12:59.437 INFO:journalctl@ceph.osd.3.smithi174.stdout:Apr 22 22:12:59 smithi174 podman[61662]: 2024-04-22 22:12:59.175428126 +0000 UTC m=+0.669246072 container died 1be3c1e8c2f42b2021e30d9f593a6adb7b97999403c880a565fb320601a950ec (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-3-deactivate, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, ceph=True, vendor=Red Hat, Inc., name=centos-stream, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public) 2024-04-22T22:12:59.563 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:12:59 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:12:59.286+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:12:59.564 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:12:59 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:12:59.446+0000 7f59e3860700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:30.511393+0000 front 2024-04-22T22:12:30.511452+0000 (oldest deadline 2024-04-22T22:12:55.211336+0000) 2024-04-22T22:12:59.807 DEBUG:teuthology.orchestra.run.smithi174:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.3.service' 2024-04-22T22:12:59.847 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:12:59.847 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-22T22:12:59.848 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-22T22:12:59.848 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.4 2024-04-22T22:13:00.021 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:12:59 smithi192 systemd[1]: Stopping Ceph osd.4 for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:13:00.021 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:12:59 smithi192 bash[49387]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-22T22:13:00.022 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:12:59 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:12:59.743+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:00.022 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:12:59 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:12:59.707+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:00.313 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:00 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:00.251+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:13:00.314 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:00 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:13:00.046+0000 7f59e8ec9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:13:00.314 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:00 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:13:00.046+0000 7f59e8ec9700 -1 osd.4 93 *** Got signal Terminated *** 2024-04-22T22:13:00.314 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:00 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:13:00.046+0000 7f59e8ec9700 -1 osd.4 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:13:00.697 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:00 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:13:00.410+0000 7f59e3860700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:30.511393+0000 front 2024-04-22T22:12:30.511452+0000 (oldest deadline 2024-04-22T22:12:55.211336+0000) 2024-04-22T22:13:01.063 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:00 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:00.744+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:01.064 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:00 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:00.723+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:01.563 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:01 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:13:01.386+0000 7f59e3860700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:30.511393+0000 front 2024-04-22T22:12:30.511452+0000 (oldest deadline 2024-04-22T22:12:55.211336+0000) 2024-04-22T22:13:01.564 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:01 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:01.247+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:13:02.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:01 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:01.676+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:02.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:01 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:01.728+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:02.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:01 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:01.728+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:02.563 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:02 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:13:02.381+0000 7f59e3860700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:30.511393+0000 front 2024-04-22T22:12:30.511452+0000 (oldest deadline 2024-04-22T22:12:55.211336+0000) 2024-04-22T22:13:02.564 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:02 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:02.267+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:13:02.933 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:02 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:02.706+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:02.933 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:02 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:02.706+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:02.933 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:02 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:02.694+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:02.933 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:02 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:02.694+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:03.563 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:03 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:13:03.427+0000 7f59e3860700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:30.511393+0000 front 2024-04-22T22:12:30.511452+0000 (oldest deadline 2024-04-22T22:12:55.211336+0000) 2024-04-22T22:13:03.564 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:03 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:13:03.427+0000 7f59e3860700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.713384+0000 front 2024-04-22T22:12:39.713481+0000 (oldest deadline 2024-04-22T22:13:03.213272+0000) 2024-04-22T22:13:03.564 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:03 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:03.262+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:13:03.564 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:03 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:03.262+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:36.442512+0000 front 2024-04-22T22:12:36.442584+0000 (oldest deadline 2024-04-22T22:13:02.342623+0000) 2024-04-22T22:13:04.063 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:03 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:03.697+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:04.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:03 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:03.697+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:04.064 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:03 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:03.710+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:04.064 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:03 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:03.710+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:04.564 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:04 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:04.293+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:13:04.564 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:04 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:04.293+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:36.442512+0000 front 2024-04-22T22:12:36.442584+0000 (oldest deadline 2024-04-22T22:13:02.342623+0000) 2024-04-22T22:13:04.564 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:04 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:13:04.435+0000 7f59e3860700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:30.511393+0000 front 2024-04-22T22:12:30.511452+0000 (oldest deadline 2024-04-22T22:12:55.211336+0000) 2024-04-22T22:13:04.564 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:04 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4[27536]: 2024-04-22T22:13:04.435+0000 7f59e3860700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.713384+0000 front 2024-04-22T22:12:39.713481+0000 (oldest deadline 2024-04-22T22:13:03.213272+0000) 2024-04-22T22:13:05.063 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:04 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:04.741+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:05.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:04 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:04.741+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:05.064 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:04 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:04.716+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:05.064 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:04 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:04.716+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:05.564 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:05 smithi192 podman[49400]: 2024-04-22 22:13:05.39284542 +0000 UTC m=+5.432070904 container died c77c665e8b26ee143b7f9452a9768c4a8fd923b88a0bd2b01ce160489e85c08f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, vcs-type=git, name=centos-stream, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.buildah.version=1.19.8, ceph=True, GIT_CLEAN=True, release=754, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux ) 2024-04-22T22:13:05.564 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:05 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:05.316+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:13:05.564 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:05 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:05.316+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:36.442512+0000 front 2024-04-22T22:12:36.442584+0000 (oldest deadline 2024-04-22T22:13:02.342623+0000) 2024-04-22T22:13:05.894 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:05 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:05.733+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:05.895 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:05 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:05.733+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:05.895 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:05 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:05.697+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:05.895 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:05 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:05.697+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:05.896 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:05 smithi192 podman[49400]: 2024-04-22 22:13:05.736696653 +0000 UTC m=+5.775922134 container cleanup c77c665e8b26ee143b7f9452a9768c4a8fd923b88a0bd2b01ce160489e85c08f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4, vcs-type=git, name=centos-stream, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, ceph=True, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, architecture=x86_64, distribution-scope=public) 2024-04-22T22:13:05.896 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:05 smithi192 bash[49400]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4 2024-04-22T22:13:06.162 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:05 smithi192 podman[49456]: 2024-04-22 22:13:05.921149833 +0000 UTC m=+0.528918920 container remove c77c665e8b26ee143b7f9452a9768c4a8fd923b88a0bd2b01ce160489e85c08f (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., RELEASE=HEAD, architecture=x86_64, ceph=True, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T22:13:06.163 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:05 smithi192 bash[49469]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.4" found: no such container 2024-04-22T22:13:06.163 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:06 smithi192 podman[49534]: 2024-04-22 22:13:06.074363711 +0000 UTC m=+0.017649207 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:13:06.163 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:06 smithi192 podman[49534]: 2024-04-22 22:13:06.188928777 +0000 UTC m=+0.132214248 container create 2778dace48fcfe9cc8de7a0dd06e8cbb99850a2e2be47943a0a49a6d44f55c9b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4-deactivate, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_BRANCH=HEAD, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream) 2024-04-22T22:13:06.464 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:06 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:06.327+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:13:06.465 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:06 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:06.327+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:36.442512+0000 front 2024-04-22T22:12:36.442584+0000 (oldest deadline 2024-04-22T22:13:02.342623+0000) 2024-04-22T22:13:06.465 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:06 smithi192 podman[49534]: 2024-04-22 22:13:06.42445006 +0000 UTC m=+0.367735532 container init 2778dace48fcfe9cc8de7a0dd06e8cbb99850a2e2be47943a0a49a6d44f55c9b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4-deactivate, vendor=Red Hat, Inc., vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, distribution-scope=public, io.openshift.expose-services=, release=754, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, name=centos-stream, GIT_CLEAN=True, RELEASE=HEAD, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8) 2024-04-22T22:13:06.465 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:06 smithi192 podman[49534]: 2024-04-22 22:13:06.427584409 +0000 UTC m=+0.370869884 container start 2778dace48fcfe9cc8de7a0dd06e8cbb99850a2e2be47943a0a49a6d44f55c9b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4-deactivate, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, maintainer=Guillaume Abrioux , architecture=x86_64, release=754, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, distribution-scope=public, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-22T22:13:06.726 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:06 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:06.665+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:06.726 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:06 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:06.665+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:06.726 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:06 smithi192 podman[49534]: 2024-04-22 22:13:06.491162311 +0000 UTC m=+0.434447798 container attach 2778dace48fcfe9cc8de7a0dd06e8cbb99850a2e2be47943a0a49a6d44f55c9b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., release=754, GIT_BRANCH=HEAD, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, distribution-scope=public, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0) 2024-04-22T22:13:06.727 INFO:journalctl@ceph.osd.4.smithi192.stdout:Apr 22 22:13:06 smithi192 podman[49534]: 2024-04-22 22:13:06.707120195 +0000 UTC m=+0.650405683 container died 2778dace48fcfe9cc8de7a0dd06e8cbb99850a2e2be47943a0a49a6d44f55c9b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=-17.2.0, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vcs-type=git, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.openshift.expose-services=, GIT_CLEAN=True, version=8, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public) 2024-04-22T22:13:06.976 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:06 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:06.751+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:06.976 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:06 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:06.751+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:07.272 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.4.service' 2024-04-22T22:13:07.311 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:13:07.311 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-22T22:13:07.311 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-22T22:13:07.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.5 2024-04-22T22:13:07.503 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:07 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:07.330+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:13:07.503 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:07 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:07.330+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:36.442512+0000 front 2024-04-22T22:12:36.442584+0000 (oldest deadline 2024-04-22T22:13:02.342623+0000) 2024-04-22T22:13:07.503 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:07 smithi192 systemd[1]: Stopping Ceph osd.5 for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:13:07.503 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:07 smithi192 bash[49613]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-22T22:13:07.813 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:07 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:07.558+0000 7f31a5274700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:13:07.814 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:07 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:07.558+0000 7f31a5274700 -1 osd.5 93 *** Got signal Terminated *** 2024-04-22T22:13:07.814 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:07 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:07.558+0000 7f31a5274700 -1 osd.5 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:13:07.814 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:07 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:07.764+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:07.814 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:07 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:07.764+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:07.815 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:07 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:07.649+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:07.815 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:07 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:07.649+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:08.600 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:08 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:08.374+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:13:08.600 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:08 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:08.374+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:36.442512+0000 front 2024-04-22T22:12:36.442584+0000 (oldest deadline 2024-04-22T22:13:02.342623+0000) 2024-04-22T22:13:09.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:08 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:08.626+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:09.064 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:08 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:08.626+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:09.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:08 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:08.777+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:09.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:08 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:08.777+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:09.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:08 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:08.777+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.624288+0000 front 2024-04-22T22:12:47.624253+0000 (oldest deadline 2024-04-22T22:13:08.724290+0000) 2024-04-22T22:13:09.643 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:09 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:09.350+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:13:09.643 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:09 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:09.350+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:36.442512+0000 front 2024-04-22T22:12:36.442584+0000 (oldest deadline 2024-04-22T22:13:02.342623+0000) 2024-04-22T22:13:10.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:09 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:09.668+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:10.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:09 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:09.668+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:10.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:09 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:09.809+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:10.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:09 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:09.809+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:10.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:09 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:09.809+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.624288+0000 front 2024-04-22T22:12:47.624253+0000 (oldest deadline 2024-04-22T22:13:08.724290+0000) 2024-04-22T22:13:10.681 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:10 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:10.339+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:13:10.681 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:10 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:10.339+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:36.442512+0000 front 2024-04-22T22:12:36.442584+0000 (oldest deadline 2024-04-22T22:13:02.342623+0000) 2024-04-22T22:13:11.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:10 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:10.707+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:11.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:10 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:10.707+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:11.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:10 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:10.767+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:11.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:10 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:10.767+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:11.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:10 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:10.767+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.624288+0000 front 2024-04-22T22:12:47.624253+0000 (oldest deadline 2024-04-22T22:13:08.724290+0000) 2024-04-22T22:13:11.636 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:11.348+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:13:11.636 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:11.348+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:36.442512+0000 front 2024-04-22T22:12:36.442584+0000 (oldest deadline 2024-04-22T22:13:02.342623+0000) 2024-04-22T22:13:12.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:11.660+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:12.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:11.660+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:12.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:11.769+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:12.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:11.769+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:12.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:11.769+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.624288+0000 front 2024-04-22T22:12:47.624253+0000 (oldest deadline 2024-04-22T22:13:08.724290+0000) 2024-04-22T22:13:12.563 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:12.331+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:27.641598+0000 front 2024-04-22T22:12:27.641615+0000 (oldest deadline 2024-04-22T22:12:52.341704+0000) 2024-04-22T22:13:12.564 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5[30425]: 2024-04-22T22:13:12.331+0000 7f319f40a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:36.442512+0000 front 2024-04-22T22:12:36.442584+0000 (oldest deadline 2024-04-22T22:13:02.342623+0000) 2024-04-22T22:13:12.906 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:12.649+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:12.907 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:12.649+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:12.907 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:12.724+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:12.907 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:12.724+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:12.907 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:12 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:12.724+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.624288+0000 front 2024-04-22T22:12:47.624253+0000 (oldest deadline 2024-04-22T22:13:08.724290+0000) 2024-04-22T22:13:13.196 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:12 smithi192 podman[49626]: 2024-04-22 22:13:12.932932932 +0000 UTC m=+5.483357763 container died 67848ad026e085a29ad649a1f75eadf358af59baf7c518c98921132a4510cb37 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, vcs-type=git, com.redhat.component=centos-stream-container, ceph=True, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8) 2024-04-22T22:13:13.196 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:13 smithi192 podman[49626]: 2024-04-22 22:13:13.222388208 +0000 UTC m=+5.772813038 container cleanup 67848ad026e085a29ad649a1f75eadf358af59baf7c518c98921132a4510cb37 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, vcs-type=git, version=8, build-date=2022-05-03T08:36:31.336870, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, distribution-scope=public, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-22T22:13:13.563 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:13 smithi192 bash[49626]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5 2024-04-22T22:13:13.563 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:13 smithi192 bash[49668]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.5" found: no such container 2024-04-22T22:13:13.563 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:13 smithi192 podman[49654]: 2024-04-22 22:13:13.456675884 +0000 UTC m=+0.519765723 container remove 67848ad026e085a29ad649a1f75eadf358af59baf7c518c98921132a4510cb37 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5, vcs-type=git, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, ceph=True, release=754, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64) 2024-04-22T22:13:13.894 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:13 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:13.758+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:13.894 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:13 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:13.758+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:13.894 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:13 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:13.758+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.624288+0000 front 2024-04-22T22:12:47.624253+0000 (oldest deadline 2024-04-22T22:13:08.724290+0000) 2024-04-22T22:13:13.894 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:13 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:13.602+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:13.895 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:13 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:13.602+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:13.895 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:13 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:13.602+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:13.895 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:13 smithi192 podman[49738]: 2024-04-22 22:13:13.575296162 +0000 UTC m=+0.019592295 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:13:13.895 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:13 smithi192 podman[49738]: 2024-04-22 22:13:13.699254209 +0000 UTC m=+0.143550371 container create b98d8d845be31a1ea09d983021500dbc29ed582092042c65ce80cc5be8424bef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_CLEAN=True, release=754, io.k8s.display-name=CentOS Stream 8, version=8, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, vendor=Red Hat, Inc., vcs-type=git) 2024-04-22T22:13:13.895 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:13 smithi192 podman[49738]: 2024-04-22 22:13:13.920483688 +0000 UTC m=+0.364779808 container init b98d8d845be31a1ea09d983021500dbc29ed582092042c65ce80cc5be8424bef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5-deactivate, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, vcs-type=git, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, version=8, maintainer=Guillaume Abrioux , distribution-scope=public, architecture=x86_64, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, RELEASE=HEAD) 2024-04-22T22:13:14.191 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:13 smithi192 podman[49738]: 2024-04-22 22:13:13.924033807 +0000 UTC m=+0.368329928 container start b98d8d845be31a1ea09d983021500dbc29ed582092042c65ce80cc5be8424bef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5-deactivate, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, RELEASE=HEAD, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, release=754, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.openshift.expose-services=, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.component=centos-stream-container) 2024-04-22T22:13:14.191 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:14 smithi192 podman[49738]: 2024-04-22 22:13:14.020353869 +0000 UTC m=+0.464649989 container attach b98d8d845be31a1ea09d983021500dbc29ed582092042c65ce80cc5be8424bef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, build-date=2022-05-03T08:36:31.336870, release=754, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, vcs-type=git, io.k8s.display-name=CentOS Stream 8, ceph=True, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.19.8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T22:13:14.456 INFO:journalctl@ceph.osd.5.smithi192.stdout:Apr 22 22:13:14 smithi192 podman[49738]: 2024-04-22 22:13:14.218184137 +0000 UTC m=+0.662480258 container died b98d8d845be31a1ea09d983021500dbc29ed582092042c65ce80cc5be8424bef (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-5-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, release=754, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.component=centos-stream-container, version=8, vcs-type=git, architecture=x86_64, vendor=Red Hat, Inc., distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-22T22:13:14.724 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:14 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:14.620+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:14.724 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:14 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:14.620+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:14.724 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:14 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:14.620+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:14.755 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.5.service' 2024-04-22T22:13:14.792 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:13:14.792 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-22T22:13:14.793 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-22T22:13:14.793 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.6 2024-04-22T22:13:15.009 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:14 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:14.748+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:15.009 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:14 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:14.748+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:15.009 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:14 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:14.748+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.624288+0000 front 2024-04-22T22:12:47.624253+0000 (oldest deadline 2024-04-22T22:13:08.724290+0000) 2024-04-22T22:13:15.009 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:14 smithi192 systemd[1]: Stopping Ceph osd.6 for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:13:15.009 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:14 smithi192 bash[49817]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-22T22:13:15.313 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:15 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:15.033+0000 7fab84811700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:13:15.314 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:15 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:15.033+0000 7fab84811700 -1 osd.6 93 *** Got signal Terminated *** 2024-04-22T22:13:15.314 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:15 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:15.033+0000 7fab84811700 -1 osd.6 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:13:16.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:15 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:15.624+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:16.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:15 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:15.624+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:16.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:15 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:15.624+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:16.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:15 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:15.720+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:16.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:15 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:15.720+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:16.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:15 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:15.720+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.624288+0000 front 2024-04-22T22:12:47.624253+0000 (oldest deadline 2024-04-22T22:13:08.724290+0000) 2024-04-22T22:13:17.063 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:16 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:16.708+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:17.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:16 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:16.708+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:17.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:16 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:16.708+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.624288+0000 front 2024-04-22T22:12:47.624253+0000 (oldest deadline 2024-04-22T22:13:08.724290+0000) 2024-04-22T22:13:17.064 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:16 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:16.617+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:17.064 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:16 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:16.617+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:17.064 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:16 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:16.617+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:17.901 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:17 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:17.671+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:17.901 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:17 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:17.671+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:17.902 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:17 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:17.671+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.624288+0000 front 2024-04-22T22:12:47.624253+0000 (oldest deadline 2024-04-22T22:13:08.724290+0000) 2024-04-22T22:13:17.902 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:17 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:17.640+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:17.902 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:17 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:17.640+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:17.902 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:17 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:17.640+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:18.736 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:18 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:18.709+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:18.737 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:18 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:18.710+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:18.737 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:18 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:18.710+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.624288+0000 front 2024-04-22T22:12:47.624253+0000 (oldest deadline 2024-04-22T22:13:08.724290+0000) 2024-04-22T22:13:18.737 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:18 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:18.605+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:18.737 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:18 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:18.605+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:18.738 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:18 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:18.605+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:18.738 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:18 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:18.605+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6830 osd.3 since back 2024-04-22T22:12:55.463256+0000 front 2024-04-22T22:12:55.463315+0000 (oldest deadline 2024-04-22T22:13:18.263582+0000) 2024-04-22T22:13:19.813 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:19 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:19.734+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:29.621465+0000 front 2024-04-22T22:12:29.621391+0000 (oldest deadline 2024-04-22T22:12:54.921425+0000) 2024-04-22T22:13:19.814 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:19 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:19.734+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.522844+0000 front 2024-04-22T22:12:39.522931+0000 (oldest deadline 2024-04-22T22:13:01.222881+0000) 2024-04-22T22:13:19.814 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:19 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6[33337]: 2024-04-22T22:13:19.734+0000 7fab7f1a8700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.624288+0000 front 2024-04-22T22:12:47.624253+0000 (oldest deadline 2024-04-22T22:13:08.724290+0000) 2024-04-22T22:13:19.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:19 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:19.558+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:19.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:19 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:19.558+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:19.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:19 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:19.558+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:19.815 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:19 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:19.558+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6830 osd.3 since back 2024-04-22T22:12:55.463256+0000 front 2024-04-22T22:12:55.463315+0000 (oldest deadline 2024-04-22T22:13:18.263582+0000) 2024-04-22T22:13:20.746 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:20 smithi192 podman[49830]: 2024-04-22 22:13:20.503229693 +0000 UTC m=+5.572294012 container died d75a88722468096b0ead4920477ec9ed852219eea7727cebb9bcddf20c21078e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_CLEAN=True, ceph=True, name=centos-stream, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , architecture=x86_64, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, release=754, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T22:13:20.747 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:20 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:20.520+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:20.747 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:20 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:20.520+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:20.747 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:20 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:20.520+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:20.747 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:20 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:20.520+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6830 osd.3 since back 2024-04-22T22:12:55.463256+0000 front 2024-04-22T22:12:55.463315+0000 (oldest deadline 2024-04-22T22:13:18.263582+0000) 2024-04-22T22:13:21.063 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:20 smithi192 podman[49830]: 2024-04-22 22:13:20.922657852 +0000 UTC m=+5.991722175 container cleanup d75a88722468096b0ead4920477ec9ed852219eea7727cebb9bcddf20c21078e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6, com.redhat.component=centos-stream-container, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, version=8, io.openshift.tags=base centos centos-stream, vcs-type=git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, io.buildah.version=1.19.8, vendor=Red Hat, Inc., ceph=True, name=centos-stream, RELEASE=HEAD, release=754, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T22:13:21.063 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:20 smithi192 bash[49830]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6 2024-04-22T22:13:21.064 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:20 smithi192 bash[50518]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.6" found: no such container 2024-04-22T22:13:21.446 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:21 smithi192 podman[50503]: 2024-04-22 22:13:21.13221315 +0000 UTC m=+0.626250393 container remove d75a88722468096b0ead4920477ec9ed852219eea7727cebb9bcddf20c21078e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, distribution-scope=public, GIT_CLEAN=True, name=centos-stream, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, com.redhat.component=centos-stream-container) 2024-04-22T22:13:21.447 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:21 smithi192 podman[50583]: 2024-04-22 22:13:21.255385096 +0000 UTC m=+0.018839883 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:13:21.447 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:21 smithi192 podman[50583]: 2024-04-22 22:13:21.366334202 +0000 UTC m=+0.129789056 container create 871fab694cc15f32ddf73631f5a4685f115b116925aca09466ad75585219cefc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6-deactivate, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, GIT_CLEAN=True, vcs-type=git, distribution-scope=public, release=754, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream) 2024-04-22T22:13:21.805 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:21 smithi192 podman[50583]: 2024-04-22 22:13:21.552686388 +0000 UTC m=+0.316141182 container init 871fab694cc15f32ddf73631f5a4685f115b116925aca09466ad75585219cefc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6-deactivate, vcs-type=git, distribution-scope=public, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, release=754, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., GIT_CLEAN=True, architecture=x86_64, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, com.redhat.component=centos-stream-container, version=8) 2024-04-22T22:13:21.806 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:21 smithi192 podman[50583]: 2024-04-22 22:13:21.556077591 +0000 UTC m=+0.319532384 container start 871fab694cc15f32ddf73631f5a4685f115b116925aca09466ad75585219cefc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.openshift.tags=base centos centos-stream, ceph=True, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, architecture=x86_64, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, GIT_CLEAN=True, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, vendor=Red Hat, Inc., version=8) 2024-04-22T22:13:21.806 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:21 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:21.513+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:21.806 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:21 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:21.513+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:21.806 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:21 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:21.513+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:21.807 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:21 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:21.513+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6830 osd.3 since back 2024-04-22T22:12:55.463256+0000 front 2024-04-22T22:12:55.463315+0000 (oldest deadline 2024-04-22T22:13:18.263582+0000) 2024-04-22T22:13:21.807 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:21 smithi192 podman[50583]: 2024-04-22 22:13:21.635711584 +0000 UTC m=+0.399166373 container attach 871fab694cc15f32ddf73631f5a4685f115b116925aca09466ad75585219cefc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, version=8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.buildah.version=1.19.8, vendor=Red Hat, Inc., architecture=x86_64, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, com.redhat.component=centos-stream-container, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True) 2024-04-22T22:13:21.807 INFO:journalctl@ceph.osd.6.smithi192.stdout:Apr 22 22:13:21 smithi192 podman[50583]: 2024-04-22 22:13:21.832220433 +0000 UTC m=+0.595675226 container died 871fab694cc15f32ddf73631f5a4685f115b116925aca09466ad75585219cefc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-6-deactivate, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, ceph=True, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, release=754, RELEASE=HEAD, GIT_BRANCH=HEAD, name=centos-stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux ) 2024-04-22T22:13:22.406 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.6.service' 2024-04-22T22:13:22.443 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:13:22.444 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-22T22:13:22.444 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-22T22:13:22.444 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.7 2024-04-22T22:13:22.658 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:22.470+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:22.658 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:22.470+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:22.658 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:22.470+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:22.659 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:22.470+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6830 osd.3 since back 2024-04-22T22:12:55.463256+0000 front 2024-04-22T22:12:55.463315+0000 (oldest deadline 2024-04-22T22:13:18.263582+0000) 2024-04-22T22:13:22.659 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:22 smithi192 systemd[1]: Stopping Ceph osd.7 for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:13:22.659 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:22 smithi192 bash[50661]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-22T22:13:22.659 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:22.683+0000 7f7f2acf9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-22T22:13:22.659 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:22.683+0000 7f7f2acf9700 -1 osd.7 93 *** Got signal Terminated *** 2024-04-22T22:13:22.659 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:22 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:22.683+0000 7f7f2acf9700 -1 osd.7 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-22T22:13:23.813 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:23 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:23.518+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:23.813 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:23 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:23.518+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:23.813 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:23 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:23.518+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:23.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:23 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:23.518+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6830 osd.3 since back 2024-04-22T22:12:55.463256+0000 front 2024-04-22T22:12:55.463315+0000 (oldest deadline 2024-04-22T22:13:18.263582+0000) 2024-04-22T22:13:24.813 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:24 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:24.516+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:24.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:24 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:24.516+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:24.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:24 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:24.516+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:24.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:24 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:24.516+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6830 osd.3 since back 2024-04-22T22:12:55.463256+0000 front 2024-04-22T22:12:55.463315+0000 (oldest deadline 2024-04-22T22:13:18.263582+0000) 2024-04-22T22:13:25.813 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:25 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:25.504+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:25.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:25 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:25.504+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:25.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:25 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:25.504+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:25.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:25 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:25.504+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6830 osd.3 since back 2024-04-22T22:12:55.463256+0000 front 2024-04-22T22:12:55.463315+0000 (oldest deadline 2024-04-22T22:13:18.263582+0000) 2024-04-22T22:13:26.813 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:26.537+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:26.813 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:26.537+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:26.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:26.537+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:26.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:26 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:26.537+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6830 osd.3 since back 2024-04-22T22:12:55.463256+0000 front 2024-04-22T22:12:55.463315+0000 (oldest deadline 2024-04-22T22:13:18.263582+0000) 2024-04-22T22:13:27.813 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:27 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:27.519+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6806 osd.0 since back 2024-04-22T22:12:28.858871+0000 front 2024-04-22T22:12:28.858715+0000 (oldest deadline 2024-04-22T22:12:53.558823+0000) 2024-04-22T22:13:27.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:27 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:27.519+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6814 osd.1 since back 2024-04-22T22:12:39.360025+0000 front 2024-04-22T22:12:39.360171+0000 (oldest deadline 2024-04-22T22:13:02.260013+0000) 2024-04-22T22:13:27.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:27 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:27.519+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6822 osd.2 since back 2024-04-22T22:12:47.962123+0000 front 2024-04-22T22:12:47.962085+0000 (oldest deadline 2024-04-22T22:13:13.262083+0000) 2024-04-22T22:13:27.814 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:27 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7[36255]: 2024-04-22T22:13:27.519+0000 7f7f27e95700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.174:6830 osd.3 since back 2024-04-22T22:12:55.463256+0000 front 2024-04-22T22:12:55.463315+0000 (oldest deadline 2024-04-22T22:13:18.263582+0000) 2024-04-22T22:13:28.313 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:28 smithi192 podman[50671]: 2024-04-22 22:13:28.064704669 +0000 UTC m=+5.487777565 container died a4170e31930d78672e8e1347176fbf4f1300b90a9a2b63018da8a28126c4709b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7, RELEASE=HEAD, GIT_BRANCH=HEAD, architecture=x86_64, com.redhat.component=centos-stream-container, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, name=centos-stream, release=754, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git) 2024-04-22T22:13:28.764 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:28 smithi192 podman[50671]: 2024-04-22 22:13:28.539770906 +0000 UTC m=+5.962843773 container cleanup a4170e31930d78672e8e1347176fbf4f1300b90a9a2b63018da8a28126c4709b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7, RELEASE=HEAD, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, GIT_CLEAN=True, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, release=754, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, vendor=Red Hat, Inc.) 2024-04-22T22:13:28.765 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:28 smithi192 bash[50671]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7 2024-04-22T22:13:28.765 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:28 smithi192 bash[50752]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd.7" found: no such container 2024-04-22T22:13:29.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:28 smithi192 podman[50739]: 2024-04-22 22:13:28.790923616 +0000 UTC m=+0.718410707 container remove a4170e31930d78672e8e1347176fbf4f1300b90a9a2b63018da8a28126c4709b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, io.openshift.expose-services=, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, release=754, architecture=x86_64, distribution-scope=public, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , version=8, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, vcs-type=git) 2024-04-22T22:13:29.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:28 smithi192 podman[50815]: 2024-04-22 22:13:28.983231422 +0000 UTC m=+0.111256661 container create 620c636b041bb2eea3ee757deae4f268feaa4aa049dd241caf9f9ba7bce40472 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, vcs-type=git, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, release=754, RELEASE=HEAD, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-22T22:13:29.063 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:28 smithi192 podman[50815]: 2024-04-22 22:13:28.891648615 +0000 UTC m=+0.019673869 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-22T22:13:29.425 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:29 smithi192 podman[50815]: 2024-04-22 22:13:29.168634202 +0000 UTC m=+0.296659443 container init 620c636b041bb2eea3ee757deae4f268feaa4aa049dd241caf9f9ba7bce40472 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7-deactivate, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, vcs-type=git, architecture=x86_64, version=8, io.buildah.version=1.19.8, release=754, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, name=centos-stream, ceph=True) 2024-04-22T22:13:29.425 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:29 smithi192 podman[50815]: 2024-04-22 22:13:29.171736064 +0000 UTC m=+0.299761307 container start 620c636b041bb2eea3ee757deae4f268feaa4aa049dd241caf9f9ba7bce40472 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7-deactivate, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, distribution-scope=public, GIT_CLEAN=True, release=754, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, name=centos-stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD) 2024-04-22T22:13:29.425 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:29 smithi192 podman[50815]: 2024-04-22 22:13:29.252276221 +0000 UTC m=+0.380301476 container attach 620c636b041bb2eea3ee757deae4f268feaa4aa049dd241caf9f9ba7bce40472 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7-deactivate, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.k8s.display-name=CentOS Stream 8, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , version=8, vcs-type=git, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, GIT_CLEAN=True) 2024-04-22T22:13:29.711 INFO:journalctl@ceph.osd.7.smithi192.stdout:Apr 22 22:13:29 smithi192 podman[50815]: 2024-04-22 22:13:29.451218513 +0000 UTC m=+0.579243753 container died 620c636b041bb2eea3ee757deae4f268feaa4aa049dd241caf9f9ba7bce40472 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=HEAD, version=8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, vcs-type=git, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream) 2024-04-22T22:13:30.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@osd.7.service' 2024-04-22T22:13:30.053 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:13:30.053 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-22T22:13:30.053 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-22T22:13:30.054 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@prometheus.a 2024-04-22T22:13:30.273 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 systemd[1]: Stopping Ceph prometheus.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:13:30.273 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 bash[50894]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-22T22:13:30.540 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:13:30.299Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-22T22:13:30.541 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:13:30.299Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-22T22:13:30.541 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:13:30.299Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-22T22:13:30.541 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:13:30.299Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-22T22:13:30.541 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:13:30.299Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-22T22:13:30.541 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:13:30.299Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-22T22:13:30.541 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:13:30.299Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-22T22:13:30.541 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:13:30.299Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-22T22:13:30.541 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:13:30.299Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-22T22:13:30.542 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:13:30.300Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-22T22:13:30.542 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:13:30.300Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-22T22:13:30.542 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a[42081]: ts=2024-04-22T22:13:30.300Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-22T22:13:30.542 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 podman[50906]: 2024-04-22 22:13:30.310059986 +0000 UTC m=+0.115199160 container died dfa924a46ee2d26e0516c8747b982f729089d1559bcc8fde454576be9f638793 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T22:13:30.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 podman[50906]: 2024-04-22 22:13:30.67734732 +0000 UTC m=+0.482486554 container cleanup dfa924a46ee2d26e0516c8747b982f729089d1559bcc8fde454576be9f638793 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-22T22:13:30.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 bash[50906]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus-a 2024-04-22T22:13:30.813 INFO:journalctl@ceph.prometheus.a.smithi192.stdout:Apr 22 22:13:30 smithi192 bash[50932]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-prometheus.a" found: no such container 2024-04-22T22:13:30.913 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@prometheus.a.service' 2024-04-22T22:13:30.947 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-22T22:13:30.947 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-22T22:13:30.947 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T22:13:32.981 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:13:32 smithi174 systemd[1]: Stopping Ceph node-exporter.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:13:32.981 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:13:32 smithi174 bash[62886]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter.a" found: no such container 2024-04-22T22:13:33.236 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:13:33 smithi174 podman[62899]: 2024-04-22 22:13:33.006794782 +0000 UTC m=+0.105501815 container died 5e1c68495a2c40c96d792f0c87600e32fa9f4c94efa21032c3330193029bc3f3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-22T22:13:33.563 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:13:33 smithi174 podman[62899]: 2024-04-22 22:13:33.446010069 +0000 UTC m=+0.544717100 container cleanup 5e1c68495a2c40c96d792f0c87600e32fa9f4c94efa21032c3330193029bc3f3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-22T22:13:33.563 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:13:33 smithi174 bash[62899]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a 2024-04-22T22:13:33.874 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:13:33 smithi174 podman[62913]: 2024-04-22 22:13:33.647499557 +0000 UTC m=+0.644759785 container remove 5e1c68495a2c40c96d792f0c87600e32fa9f4c94efa21032c3330193029bc3f3 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-22T22:13:33.874 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:13:33 smithi174 systemd[1]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-22T22:13:33.874 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:13:33 smithi174 bash[62927]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter.a" found: no such container 2024-04-22T22:13:33.875 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:13:33 smithi174 systemd[1]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-22T22:13:33.875 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:13:33 smithi174 systemd[1]: Stopped Ceph node-exporter.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:13:33.875 INFO:journalctl@ceph.node-exporter.a.smithi174.stdout:Apr 22 22:13:33 smithi174 systemd[1]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@node-exporter.a.service: Consumed 3.122s CPU time. 2024-04-22T22:13:34.313 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:13:33 smithi174 systemd[1]: Stopping Ceph alertmanager.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:13:34.313 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:13:33 smithi174 bash[62979]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-22T22:13:34.314 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:13:34 smithi174 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a[47598]: level=info ts=2024-04-22T22:13:34.076Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-22T22:13:34.314 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:13:34 smithi174 podman[62992]: 2024-04-22 22:13:34.127382306 +0000 UTC m=+0.196264558 container died 1e9de7a331d8e2f1ed2c169d855595d5245cdb6d3298a8b42c3f89debd3b8bb6 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T22:13:34.626 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:13:34 smithi174 podman[62992]: 2024-04-22 22:13:34.467270712 +0000 UTC m=+0.536152961 container cleanup 1e9de7a331d8e2f1ed2c169d855595d5245cdb6d3298a8b42c3f89debd3b8bb6 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T22:13:34.627 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:13:34 smithi174 bash[62992]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a 2024-04-22T22:13:34.994 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:13:34 smithi174 podman[63006]: 2024-04-22 22:13:34.651467995 +0000 UTC m=+0.516466836 container remove 1e9de7a331d8e2f1ed2c169d855595d5245cdb6d3298a8b42c3f89debd3b8bb6 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-22T22:13:34.995 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:13:34 smithi174 podman[63006]: 2024-04-22 22:13:34.727412811 +0000 UTC m=+0.592411608 volume remove 93fa0454b17ff2f97634cfaa9f8620f0fb619ea685ad5623b23a6c7eb8a0eff2 2024-04-22T22:13:34.995 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:13:34 smithi174 bash[63018]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-alertmanager.a" found: no such container 2024-04-22T22:13:34.995 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:13:34 smithi174 systemd[1]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-22T22:13:34.995 INFO:journalctl@ceph.alertmanager.a.smithi174.stdout:Apr 22 22:13:34 smithi174 systemd[1]: Stopped Ceph alertmanager.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:14:09.100 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-22T22:14:10.880 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:14:10 smithi192 systemd[1]: Stopping Ceph node-exporter.b for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:14:10.880 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:14:10 smithi192 bash[52113]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter.b" found: no such container 2024-04-22T22:14:11.160 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:14:10 smithi192 podman[52126]: 2024-04-22 22:14:10.905997881 +0000 UTC m=+0.111493558 container died 71966acd94b16a39818e38cf5a6d06c8c1475c162048cad096241c4f9a708790 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-22T22:14:11.160 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:14:11 smithi192 podman[52126]: 2024-04-22 22:14:11.18699382 +0000 UTC m=+0.392489545 container cleanup 71966acd94b16a39818e38cf5a6d06c8c1475c162048cad096241c4f9a708790 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-22T22:14:11.467 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:14:11 smithi192 bash[52126]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b 2024-04-22T22:14:11.467 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:14:11 smithi192 podman[52140]: 2024-04-22 22:14:11.304464215 +0000 UTC m=+0.402481522 container remove 71966acd94b16a39818e38cf5a6d06c8c1475c162048cad096241c4f9a708790 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-22T22:14:11.467 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:14:11 smithi192 systemd[1]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-22T22:14:11.467 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:14:11 smithi192 bash[52156]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-node-exporter.b" found: no such container 2024-04-22T22:14:11.468 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:14:11 smithi192 systemd[1]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-22T22:14:11.468 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:14:11 smithi192 systemd[1]: Stopped Ceph node-exporter.b for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:14:11.468 INFO:journalctl@ceph.node-exporter.b.smithi192.stdout:Apr 22 22:14:11 smithi192 systemd[1]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@node-exporter.b.service: Consumed 3.068s CPU time. 2024-04-22T22:14:11.813 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:14:11 smithi192 systemd[1]: Stopping Ceph grafana.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968... 2024-04-22T22:14:11.814 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:14:11 smithi192 bash[52205]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana.a" found: no such container 2024-04-22T22:14:11.814 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:14:11 smithi192 ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a[41132]: t=2024-04-22T22:14:11+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-04-22T22:14:11.814 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:14:11 smithi192 podman[52217]: 2024-04-22 22:14:11.651697945 +0000 UTC m=+0.108074778 container died 4787c8784d2e7df5eb83443bf96fde8c36e9814cca3de13ab9981cc3e6015421 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a, vendor=Red Hat, Inc., description=Ceph Grafana Container, distribution-scope=public, com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.expose-services=, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, build-date=2022-03-28T10:36:18.413762, release=236.1648460182, name=ubi8, vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.buildah.version=1.24.2, io.openshift.tags=base rhel8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-22T22:14:12.134 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:14:11 smithi192 podman[52217]: 2024-04-22 22:14:11.97423245 +0000 UTC m=+0.430609284 container cleanup 4787c8784d2e7df5eb83443bf96fde8c36e9814cca3de13ab9981cc3e6015421 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.24.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=236.1648460182, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, distribution-scope=public, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=ubi8, vcs-type=git, io.openshift.expose-services=, vendor=Red Hat, Inc.) 2024-04-22T22:14:12.134 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:14:11 smithi192 bash[52217]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a 2024-04-22T22:14:12.135 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:14:11 smithi192 bash[52244]: Error: no container with name or ID "ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana.a" found: no such container 2024-04-22T22:14:12.135 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:14:12 smithi192 podman[52231]: 2024-04-22 22:14:12.141915822 +0000 UTC m=+0.486808505 container remove 4787c8784d2e7df5eb83443bf96fde8c36e9814cca3de13ab9981cc3e6015421 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, build-date=2022-03-28T10:36:18.413762, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, description=Ceph Grafana Container, name=ubi8, release=236.1648460182) 2024-04-22T22:14:12.135 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:14:12 smithi192 systemd[1]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-22T22:14:12.135 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:14:12 smithi192 systemd[1]: Stopped Ceph grafana.a for b7cbfdbc-00f3-11ef-bc93-c7b262605968. 2024-04-22T22:14:12.135 INFO:journalctl@ceph.grafana.a.smithi192.stdout:Apr 22 22:14:12 smithi192 systemd[1]: ceph-b7cbfdbc-00f3-11ef-bc93-c7b262605968@grafana.a.service: Consumed 2.936s CPU time. 2024-04-22T22:14:34.575 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:14:34.607 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-22T22:14:34.636 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-22T22:14:34.638 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668811/remote/smithi174/crash 2024-04-22T22:14:34.639 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/crash -- . 2024-04-22T22:14:34.712 INFO:teuthology.orchestra.run.smithi174.stderr:tar: /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T22:14:34.712 INFO:teuthology.orchestra.run.smithi174.stderr:tar: Error is not recoverable: exiting now 2024-04-22T22:14:34.714 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668811/remote/smithi192/crash 2024-04-22T22:14:34.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/crash -- . 2024-04-22T22:14:34.793 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-22T22:14:34.793 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now 2024-04-22T22:14:34.798 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-22T22:14:34.798 DEBUG:teuthology.orchestra.run.smithi174:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-22T22:14:34.828 INFO:teuthology.orchestra.run.smithi174.stdout:2024-04-22T22:03:44.632442+0000 mon.a (mon.0) 372 : cluster [WRN] Health check failed: 1 stray daemon(s) not managed by cephadm (CEPHADM_STRAY_DAEMON) 2024-04-22T22:14:34.828 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-22T22:14:34.828 INFO:tasks.cephadm:Compressing logs... 2024-04-22T22:14:34.829 DEBUG:teuthology.orchestra.run.smithi174:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T22:14:34.875 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-22T22:14:34.895 INFO:teuthology.orchestra.run.smithi174.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.log: No such file or directory 2024-04-22T22:14:34.895 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-22T22:14:34.895 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi174.ggrvao.log 2024-04-22T22:14:34.899 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-22T22:14:34.899 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi174.ggrvao.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T22:14:34.899 INFO:teuthology.orchestra.run.smithi174.stderr: 62.5% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi174.ggrvao.log.gz 2024-04-22T22:14:34.899 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi174.eqkeoc.log 2024-04-22T22:14:34.899 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-22T22:14:34.899 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.audit.log: /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi174.eqkeoc.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-22T22:14:34.899 INFO:teuthology.orchestra.run.smithi174.stderr: 62.5% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi174.eqkeoc.log.gz 2024-04-22T22:14:34.900 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-22T22:14:34.900 INFO:teuthology.orchestra.run.smithi174.stderr:gzip/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.1.log: 92.9% -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-22T22:14:34.900 INFO:teuthology.orchestra.run.smithi174.stderr: -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T22:14:34.900 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-22T22:14:34.900 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-22T22:14:34.907 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mon.c.log: /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.0.log: 94.1% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T22:14:34.907 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T22:14:34.907 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/tcmu-runner.log 2024-04-22T22:14:34.908 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.cephadm.log: 89.0% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T22:14:34.908 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T22:14:34.908 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/tcmu-runner.log: 66.4% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/tcmu-runner.log.gz 2024-04-22T22:14:34.908 INFO:teuthology.orchestra.run.smithi174.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T22:14:34.911 INFO:teuthology.orchestra.run.smithi192.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.log 2024-04-22T22:14:34.911 INFO:teuthology.orchestra.run.smithi192.stderr:: No such file or directory 2024-04-22T22:14:34.911 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-22T22:14:34.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-22T22:14:34.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-22T22:14:34.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-22T22:14:34.911 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-22T22:14:34.912 INFO:teuthology.orchestra.run.smithi192.stderr: 87.5% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-22T22:14:34.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-22T22:14:34.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-22T22:14:34.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi192.nzyxdu.log 2024-04-22T22:14:34.912 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.cephadm.log: 90.5%gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi192.ihnxdd.log 2024-04-22T22:14:34.912 INFO:teuthology.orchestra.run.smithi192.stderr: -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-22T22:14:34.913 INFO:teuthology.orchestra.run.smithi192.stderr: 81.0% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-22T22:14:34.914 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi192.nzyxdu.log: 62.5% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-client.rgw.smpl.smithi192.nzyxdu.log.gz 2024-04-22T22:14:34.914 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-22T22:14:34.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi192.ihnxdd.log: gzip -5 --verbose -- /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-22T22:14:34.915 INFO:teuthology.orchestra.run.smithi192.stderr: 62.5% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-client.rgw.foo.smithi192.ihnxdd.log.gz/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-volume.log: 2024-04-22T22:14:34.915 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-22T22:14:34.915 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 91.2% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-22T22:14:34.918 INFO:teuthology.orchestra.run.smithi192.stderr: 91.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T22:14:34.920 INFO:teuthology.orchestra.run.smithi192.stderr: 92.1% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T22:14:34.970 INFO:teuthology.orchestra.run.smithi174.stderr:/var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/cephadm.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-22T22:14:34.987 INFO:teuthology.orchestra.run.smithi174.stderr: 92.1% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-22T22:14:35.106 INFO:teuthology.orchestra.run.smithi174.stderr: 87.7% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-22T22:14:35.123 INFO:teuthology.orchestra.run.smithi192.stderr: 92.3% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-22T22:14:35.125 INFO:teuthology.orchestra.run.smithi174.stderr: 92.3% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-22T22:14:35.206 INFO:teuthology.orchestra.run.smithi192.stderr: 93.2% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-22T22:14:35.276 INFO:teuthology.orchestra.run.smithi174.stderr: 93.1% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-22T22:14:35.300 INFO:teuthology.orchestra.run.smithi192.stderr: 93.4% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-22T22:14:35.330 INFO:teuthology.orchestra.run.smithi174.stderr: 93.5% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-22T22:14:35.351 INFO:teuthology.orchestra.run.smithi192.stderr: 93.6% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-22T22:14:35.355 INFO:teuthology.orchestra.run.smithi192.stderr: 93.8% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-22T22:14:35.356 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-04-22T22:14:35.357 INFO:teuthology.orchestra.run.smithi192.stderr:real 0m0.467s 2024-04-22T22:14:35.357 INFO:teuthology.orchestra.run.smithi192.stderr:user 0m1.812s 2024-04-22T22:14:35.357 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m0.089s 2024-04-22T22:14:35.383 INFO:teuthology.orchestra.run.smithi174.stderr: 91.1% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-22T22:14:35.452 INFO:teuthology.orchestra.run.smithi174.stderr: 93.5% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-22T22:14:35.497 INFO:teuthology.orchestra.run.smithi174.stderr: 93.6% -- replaced with /var/log/ceph/b7cbfdbc-00f3-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-22T22:14:35.498 INFO:teuthology.orchestra.run.smithi174.stderr: 2024-04-22T22:14:35.499 INFO:teuthology.orchestra.run.smithi174.stderr:real 0m0.614s 2024-04-22T22:14:35.499 INFO:teuthology.orchestra.run.smithi174.stderr:user 0m2.790s 2024-04-22T22:14:35.499 INFO:teuthology.orchestra.run.smithi174.stderr:sys 0m0.127s 2024-04-22T22:14:35.499 INFO:tasks.cephadm:Archiving logs... 2024-04-22T22:14:35.499 DEBUG:teuthology.misc:Transferring archived files from smithi174:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668811/remote/smithi174/log 2024-04-22T22:14:35.500 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T22:14:35.979 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668811/remote/smithi192/log 2024-04-22T22:14:35.980 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-22T22:14:36.319 INFO:tasks.cephadm:Removing cluster... 2024-04-22T22:14:36.320 DEBUG:teuthology.orchestra.run.smithi174:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 --force 2024-04-22T22:14:36.593 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 --force 2024-04-22T22:14:36.863 INFO:tasks.cephadm:Removing cephadm ... 2024-04-22T22:14:36.863 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T22:14:36.879 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-22T22:14:36.894 INFO:tasks.cephadm:Teardown complete 2024-04-22T22:14:36.895 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-22T22:14:36.906 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-22T22:14:36.907 DEBUG:teuthology.orchestra.run.smithi174:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T22:14:36.926 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-22T22:14:36.936 INFO:teuthology.orchestra.run.smithi174.stderr:bash: line 1: ntpq: command not found 2024-04-22T22:14:36.940 INFO:teuthology.orchestra.run.smithi174.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T22:14:36.940 INFO:teuthology.orchestra.run.smithi174.stdout:=============================================================================== 2024-04-22T22:14:36.941 INFO:teuthology.orchestra.run.smithi174.stdout:^- hv01.front.sepia.ceph.com 2 6 377 7 +1102us[+1102us] +/- 583ms 2024-04-22T22:14:36.941 INFO:teuthology.orchestra.run.smithi174.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 -84us[ -79us] +/- 65ms 2024-04-22T22:14:36.941 INFO:teuthology.orchestra.run.smithi174.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +132us[ +132us] +/- 61ms 2024-04-22T22:14:36.941 INFO:teuthology.orchestra.run.smithi174.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:14:36.948 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-04-22T22:14:36.951 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-22T22:14:36.952 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-04-22T22:14:36.952 INFO:teuthology.orchestra.run.smithi192.stdout:^- hv01.front.sepia.ceph.com 2 6 377 5 +1119us[+1119us] +/- 583ms 2024-04-22T22:14:36.952 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -73us[ -67us] +/- 65ms 2024-04-22T22:14:36.952 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 5 +153us[ +153us] +/- 61ms 2024-04-22T22:14:36.952 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-22T22:14:36.953 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-22T22:14:36.963 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-22T22:14:36.963 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-22T22:14:36.973 DEBUG:teuthology.orchestra.run.smithi174:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T22:14:37.050 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-22T22:14:37.111 DEBUG:teuthology.orchestra.run.smithi174:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T22:14:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:14:37.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-22T22:14:37.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:14:37.165 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-22T22:14:37.177 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-22T22:14:37.186 INFO:teuthology.task.internal:Duration was 1066.571049 seconds 2024-04-22T22:14:37.187 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-22T22:14:37.195 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-22T22:14:37.196 DEBUG:teuthology.orchestra.run.smithi174:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T22:14:37.198 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-22T22:14:37.233 INFO:teuthology.orchestra.run.smithi174.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T22:14:37.241 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-22T22:14:37.600 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-22T22:14:37.601 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi174.front.sepia.ceph.com 2024-04-22T22:14:37.601 DEBUG:teuthology.orchestra.run.smithi174:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T22:14:37.621 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-04-22T22:14:37.622 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-22T22:14:37.662 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-22T22:14:37.662 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T22:14:37.664 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-22T22:14:37.727 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-22T22:14:37.727 DEBUG:teuthology.orchestra.run.smithi174:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T22:14:37.729 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-22T22:14:38.050 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-22T22:14:38.060 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-22T22:14:38.061 DEBUG:teuthology.orchestra.run.smithi174:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T22:14:38.085 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-22T22:14:38.109 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-22T22:14:38.122 DEBUG:teuthology.orchestra.run.smithi174:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T22:14:38.127 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-22T22:14:38.146 INFO:teuthology.orchestra.run.smithi174.stdout:kernel.core_pattern = core 2024-04-22T22:14:38.171 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-04-22T22:14:38.200 DEBUG:teuthology.orchestra.run.smithi174:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T22:14:38.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:14:38.215 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-22T22:14:38.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-22T22:14:38.255 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-22T22:14:38.267 INFO:teuthology.task.internal:Transferring archived files... 2024-04-22T22:14:38.268 DEBUG:teuthology.misc:Transferring archived files from smithi174:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668811/remote/smithi174 2024-04-22T22:14:38.269 DEBUG:teuthology.orchestra.run.smithi174:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T22:14:38.312 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668811/remote/smithi192 2024-04-22T22:14:38.312 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-22T22:14:38.351 INFO:teuthology.task.internal:Removing archive directory... 2024-04-22T22:14:38.352 DEBUG:teuthology.orchestra.run.smithi174:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T22:14:38.354 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-22T22:14:38.399 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-22T22:14:38.411 INFO:teuthology.task.internal:Not uploading archives. 2024-04-22T22:14:38.411 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-22T22:14:38.421 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-22T22:14:38.422 DEBUG:teuthology.orchestra.run.smithi174:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T22:14:38.424 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-22T22:14:38.436 INFO:teuthology.orchestra.run.smithi174.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 22:14 /home/ubuntu/cephtest 2024-04-22T22:14:38.453 INFO:teuthology.orchestra.run.smithi192.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 22 22:14 /home/ubuntu/cephtest 2024-04-22T22:14:38.454 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-22T22:14:38.466 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-22T22:14:38.490 INFO:teuthology.nuke:Checking targets against current locks 2024-04-22T22:14:38.522 DEBUG:teuthology.nuke:shortname: smithi174 2024-04-22T22:14:38.523 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T22:14:38.547 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi174.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668811', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 21:50:16.053059', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLCmdAO+N68U0PLDLxlMe57+huFXk4h+ektZmUJNIBph9/kp1iHAT/aAZr6HprXbgEaQVy3XhXELyOlZ0BQvL+I='} 2024-04-22T22:14:38.551 DEBUG:teuthology.nuke:shortname: smithi192 2024-04-22T22:14:38.552 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-22T22:14:38.578 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-22_21:08:03-orch-squid-distro-default-smithi/7668811', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-22 21:50:16.051497', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP1XXqiS8lqW9G9jAODYdBqySY2EUZPuRew3OjNPF0/ZR2ZC9/IZu84f4Q8SZ1CmLfuoNDRpeGlrfuoXlECwxqI='} 2024-04-22T22:14:38.618 INFO:teuthology.orchestra.console.smithi174:Power off 2024-04-22T22:14:38.619 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T22:14:38.646 INFO:teuthology.orchestra.console.smithi192:Power off 2024-04-22T22:14:38.647 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-22T22:14:38.662 DEBUG:teuthology.orchestra.console.smithi174:power off output: Chassis Power Control: Down/Off 2024-04-22T22:14:38.662 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T22:14:38.684 DEBUG:teuthology.orchestra.console.smithi192:power off output: Chassis Power Control: Down/Off 2024-04-22T22:14:38.685 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T22:14:38.700 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-04-22T22:14:38.715 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-04-22T22:14:42.702 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T22:14:42.716 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T22:14:42.815 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is on 2024-04-22T22:14:42.826 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is on 2024-04-22T22:14:46.818 DEBUG:teuthology.orchestra.console.smithi174:pexpect command: ipmitool -H smithi174.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T22:14:46.843 DEBUG:teuthology.orchestra.console.smithi192:pexpect command: ipmitool -H smithi192.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-22T22:14:46.945 DEBUG:teuthology.orchestra.console.smithi174:check power output: Chassis Power is off 2024-04-22T22:14:46.964 DEBUG:teuthology.orchestra.console.smithi192:check power output: Chassis Power is off 2024-04-22T22:14:47.046 INFO:teuthology.orchestra.console.smithi174:Power off completed 2024-04-22T22:14:47.065 INFO:teuthology.orchestra.console.smithi192:Power off completed 2024-04-22T22:14:47.177 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/repo_digest 3-upgrade/simple 4-wait 5-upgrade-ls agent/on mon_election/classic} duration: 1066.5710489749908 failure_reason: 'Command failed on smithi174 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid b7cbfdbc-00f3-11ef-bc93-c7b262605968 -e sha1=430e09df97c8fc7dc2b2ae424f68ed11366c540f -- bash -c \''ceph versions | jq -e \''"\''"\''.overall | keys\''"\''"\'' | grep $sha1\''''' owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b71aab70a00a442480511c1049f2cf9d status: fail success: false 2024-04-22T22:14:47.177 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-22T22:14:47.271 INFO:teuthology.run:FAIL