2024-04-15T21:35:04.517 INFO:root:teuthology version: 0.0.1.dev267+g6c63784 2024-04-15T21:35:04.518 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657477 --verbose --owner scheduled_teuthology@teuthology --description orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} -- /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657477/orig.config.yaml 2024-04-15T21:35:04.551 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T21:35:04.632 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657477 branch: squid description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} email: ceph-qa@ceph.com first_in_suite: false job_id: '7657477' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon election default strategy: 3 mgr: debug mgr: 20 debug ms: 1 mgr/cephadm/use_agent: false mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 workunit: branch: squid sha1: 461a84e6fbbf089011eafd13b98169fb9e9b8cc9 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b seed: 7775 sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 sleep_before_teardown: 0 subset: 18/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 461a84e6fbbf089011eafd13b98169fb9e9b8cc9 targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPwmQiH618WrhD7w/h2f5W0qF3ls5QPMhWNPeYdcZr4q4iT15pBHmc0403j29s+ld3WDFLSkKpMLJpnXpD/9vb4= smithi033.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBECNfbdraZfwlGxP5ilb6Y0oCb66kvkAJqLd2/ZRoNcNbQXVxHTIbnj5O3iwaBvFljqZCLlwJCN5/sB23No0wYA= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 - cephadm.shell: env: - sha1 mon.a: - radosgw-admin realm create --rgw-realm=r --default - radosgw-admin zonegroup create --rgw-zonegroup=default --master --default - radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default - radosgw-admin period update --rgw-realm=r --commit - ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000 - ceph orch apply rgw smpl - ceph osd pool create foo - rbd pool init foo - ceph orch apply iscsi foo u p - sleep 120 - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 mon.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph orch upgrade status - ceph health detail - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - ceph orch ls | grep '^osd ' - cephadm.shell: mon.a: - ceph orch upgrade ls - ceph orch upgrade ls --image quay.io/ceph/ceph --show-all-versions | grep 16.2.0 - ceph orch upgrade ls --image quay.io/ceph/ceph --tags | grep v16.2.2 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6c637841c215537a4502385240412f1966e0faab timestamp: 2024-04-15_21:08:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-15T21:35:04.632 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa; will attempt to use it 2024-04-15T21:35:04.633 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks 2024-04-15T21:35:04.633 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-15T21:35:04.634 INFO:teuthology.task.internal:Checking packages... 2024-04-15T21:35:04.656 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a9a752df26c63acad72e1b3569fd79a515ca0765' 2024-04-15T21:35:04.656 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-15T21:35:04.657 INFO:teuthology.packaging:ref: None 2024-04-15T21:35:04.657 INFO:teuthology.packaging:tag: None 2024-04-15T21:35:04.657 INFO:teuthology.packaging:branch: squid 2024-04-15T21:35:04.657 INFO:teuthology.packaging:sha1: a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T21:35:04.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-04-15T21:35:04.824 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2114.ga9a752df 2024-04-15T21:35:04.826 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-15T21:35:04.831 INFO:teuthology.task.internal:no buildpackages task found 2024-04-15T21:35:04.831 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-15T21:35:04.840 INFO:teuthology.task.internal:Saving configuration 2024-04-15T21:35:04.851 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-15T21:35:04.858 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T21:35:04.884 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657477', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 21:30:28.093615', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPwmQiH618WrhD7w/h2f5W0qF3ls5QPMhWNPeYdcZr4q4iT15pBHmc0403j29s+ld3WDFLSkKpMLJpnXpD/9vb4='} 2024-04-15T21:35:04.907 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657477', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 21:30:28.092212', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBECNfbdraZfwlGxP5ilb6Y0oCb66kvkAJqLd2/ZRoNcNbQXVxHTIbnj5O3iwaBvFljqZCLlwJCN5/sB23No0wYA='} 2024-04-15T21:35:04.907 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-15T21:35:04.915 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a', 'alertmanager.a'] 2024-04-15T21:35:04.915 INFO:teuthology.task.internal:roles: ubuntu@smithi033.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b'] 2024-04-15T21:35:04.915 INFO:teuthology.run_tasks:Running task console_log... 2024-04-15T21:35:04.988 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7feb77aaa0d0>, signals=[15]) 2024-04-15T21:35:04.988 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-15T21:35:04.999 INFO:teuthology.task.internal:Opening connections... 2024-04-15T21:35:04.999 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-04-15T21:35:05.001 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T21:35:05.104 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi033.front.sepia.ceph.com 2024-04-15T21:35:05.105 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T21:35:05.207 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-15T21:35:05.216 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-04-15T21:35:05.233 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-04-15T21:35:05.234 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-04-15T21:35:05.290 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2024-04-15T21:35:05.290 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="9" 2024-04-15T21:35:05.290 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2024-04-15T21:35:05.290 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2024-04-15T21:35:05.290 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="9" 2024-04-15T21:35:05.290 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el9" 2024-04-15T21:35:05.290 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T21:35:05.291 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2024-04-15T21:35:05.291 INFO:teuthology.orchestra.run.smithi007.stdout:LOGO="fedora-logo-icon" 2024-04-15T21:35:05.291 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T21:35:05.291 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2024-04-15T21:35:05.291 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T21:35:05.291 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T21:35:05.291 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T21:35:05.291 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-04-15T21:35:05.309 DEBUG:teuthology.orchestra.run.smithi033:> uname -m 2024-04-15T21:35:05.327 INFO:teuthology.orchestra.run.smithi033.stdout:x86_64 2024-04-15T21:35:05.327 DEBUG:teuthology.orchestra.run.smithi033:> cat /etc/os-release 2024-04-15T21:35:05.384 INFO:teuthology.orchestra.run.smithi033.stdout:NAME="CentOS Stream" 2024-04-15T21:35:05.384 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION="9" 2024-04-15T21:35:05.384 INFO:teuthology.orchestra.run.smithi033.stdout:ID="centos" 2024-04-15T21:35:05.384 INFO:teuthology.orchestra.run.smithi033.stdout:ID_LIKE="rhel fedora" 2024-04-15T21:35:05.384 INFO:teuthology.orchestra.run.smithi033.stdout:VERSION_ID="9" 2024-04-15T21:35:05.384 INFO:teuthology.orchestra.run.smithi033.stdout:PLATFORM_ID="platform:el9" 2024-04-15T21:35:05.384 INFO:teuthology.orchestra.run.smithi033.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-15T21:35:05.384 INFO:teuthology.orchestra.run.smithi033.stdout:ANSI_COLOR="0;31" 2024-04-15T21:35:05.384 INFO:teuthology.orchestra.run.smithi033.stdout:LOGO="fedora-logo-icon" 2024-04-15T21:35:05.384 INFO:teuthology.orchestra.run.smithi033.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-15T21:35:05.384 INFO:teuthology.orchestra.run.smithi033.stdout:HOME_URL="https://centos.org/" 2024-04-15T21:35:05.384 INFO:teuthology.orchestra.run.smithi033.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-15T21:35:05.385 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-15T21:35:05.385 INFO:teuthology.orchestra.run.smithi033.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-15T21:35:05.385 INFO:teuthology.lock.ops:Updating smithi033.front.sepia.ceph.com on lock server 2024-04-15T21:35:05.409 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-15T21:35:05.422 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-15T21:35:05.434 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-15T21:35:05.435 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-04-15T21:35:05.438 DEBUG:teuthology.orchestra.run.smithi033:> test '!' -e /home/ubuntu/cephtest 2024-04-15T21:35:05.453 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-15T21:35:05.464 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-15T21:35:05.464 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-04-15T21:35:05.495 DEBUG:teuthology.orchestra.run.smithi033:> test -z $(ls -A /var/lib/ceph) 2024-04-15T21:35:05.520 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-15T21:35:05.577 INFO:teuthology.run_tasks:Running task kernel... 2024-04-15T21:35:05.597 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-04-15T21:35:05.597 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-04-15T21:35:05.598 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T21:35:05.598 DEBUG:teuthology.orchestra.run.smithi033:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-15T21:35:05.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T21:35:05.616 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-04-15T21:35:05.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T21:35:05.617 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-04-15T21:35:05.673 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-361.el9.x86_64 2024-04-15T21:35:05.673 INFO:teuthology.task.kernel:Running kernel on smithi033: 5.14.0-361.el9.x86_64 2024-04-15T21:35:05.673 DEBUG:teuthology.orchestra.run.smithi033:> sudo yum install -y kernel 2024-04-15T21:35:05.674 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-361.el9.x86_64 2024-04-15T21:35:05.675 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.14.0-361.el9.x86_64 2024-04-15T21:35:05.675 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2024-04-15T21:35:07.735 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-04-15T21:35:07.828 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-04-15T21:35:09.211 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-04-15T21:35:09.261 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-04-15T21:35:10.011 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.0 MB 00:00 2024-04-15T21:35:12.379 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-04-15T21:35:16.504 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-04-15T21:35:18.358 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - BaseOS 935 kB/s | 8.0 MB 00:08 2024-04-15T21:35:18.846 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-04-15T21:35:20.301 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-04-15T21:35:22.801 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 5.5 MB/s | 21 MB 00:03 2024-04-15T21:35:24.518 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-15T21:35:26.286 INFO:teuthology.orchestra.run.smithi033.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-04-15T21:35:27.756 INFO:teuthology.orchestra.run.smithi033.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-04-15T21:35:28.197 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-15T21:35:30.679 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T21:35:30.679 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T21:35:30.774 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-04-15T21:35:30.791 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-15T21:35:30.791 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-04-15T21:35:30.791 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-15T21:35:30.791 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-04-15T21:35:30.791 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T21:35:30.791 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-04-15T21:35:30.791 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T21:35:30.791 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T21:35:30.791 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T21:35:30.792 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:35:30.792 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-04-15T21:35:30.792 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-04-15T21:35:30.792 INFO:teuthology.orchestra.run.smithi007.stdout:Install 4 Packages 2024-04-15T21:35:30.792 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:35:30.792 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 77 M 2024-04-15T21:35:30.792 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 123 M 2024-04-15T21:35:30.792 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-04-15T21:35:30.998 INFO:teuthology.orchestra.run.smithi007.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.3 MB/s | 277 kB 00:00 2024-04-15T21:35:31.374 INFO:teuthology.orchestra.run.smithi007.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 26 MB/s | 15 MB 00:00 2024-04-15T21:35:31.966 INFO:teuthology.orchestra.run.smithi007.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 29 MB/s | 34 MB 00:01 2024-04-15T21:35:32.482 INFO:teuthology.orchestra.run.smithi007.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 19 MB/s | 28 MB 00:01 2024-04-15T21:35:32.485 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-04-15T21:35:32.486 INFO:teuthology.orchestra.run.smithi007.stdout:Total 45 MB/s | 77 MB 00:01 2024-04-15T21:35:32.486 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-04-15T21:35:32.904 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-04-15T21:35:32.904 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-04-15T21:35:32.995 INFO:teuthology.orchestra.run.smithi033.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-15T21:35:33.774 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-04-15T21:35:33.774 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-04-15T21:35:35.460 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-15T21:35:35.460 INFO:teuthology.orchestra.run.smithi033.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-15T21:35:35.551 INFO:teuthology.orchestra.run.smithi033.stdout:Dependencies resolved. 2024-04-15T21:35:35.566 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-04-15T21:35:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: Package Arch Version Repository Size 2024-04-15T21:35:35.566 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-04-15T21:35:35.566 INFO:teuthology.orchestra.run.smithi033.stdout:Installing: 2024-04-15T21:35:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: kernel x86_64 5.14.0-438.el9 CentOS-BaseOS 277 k 2024-04-15T21:35:35.566 INFO:teuthology.orchestra.run.smithi033.stdout:Installing dependencies: 2024-04-15T21:35:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core x86_64 5.14.0-438.el9 CentOS-BaseOS 15 M 2024-04-15T21:35:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules x86_64 5.14.0-438.el9 CentOS-BaseOS 34 M 2024-04-15T21:35:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core x86_64 5.14.0-438.el9 CentOS-BaseOS 28 M 2024-04-15T21:35:35.566 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-15T21:35:35.566 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction Summary 2024-04-15T21:35:35.567 INFO:teuthology.orchestra.run.smithi033.stdout:================================================================================ 2024-04-15T21:35:35.567 INFO:teuthology.orchestra.run.smithi033.stdout:Install 4 Packages 2024-04-15T21:35:35.567 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-15T21:35:35.567 INFO:teuthology.orchestra.run.smithi033.stdout:Total download size: 77 M 2024-04-15T21:35:35.567 INFO:teuthology.orchestra.run.smithi033.stdout:Installed size: 123 M 2024-04-15T21:35:35.567 INFO:teuthology.orchestra.run.smithi033.stdout:Downloading Packages: 2024-04-15T21:35:35.747 INFO:teuthology.orchestra.run.smithi033.stdout:(1/4): kernel-5.14.0-438.el9.x86_64.rpm 1.5 MB/s | 277 kB 00:00 2024-04-15T21:35:36.243 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-04-15T21:35:36.447 INFO:teuthology.orchestra.run.smithi033.stdout:(2/4): kernel-core-5.14.0-438.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-04-15T21:35:36.871 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T21:35:36.961 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T21:35:37.022 INFO:teuthology.orchestra.run.smithi033.stdout:(3/4): kernel-modules-5.14.0-438.el9.x86_64.rpm 23 MB/s | 34 MB 00:01 2024-04-15T21:35:37.364 INFO:teuthology.orchestra.run.smithi033.stdout:(4/4): kernel-modules-core-5.14.0-438.el9.x86_6 17 MB/s | 28 MB 00:01 2024-04-15T21:35:37.367 INFO:teuthology.orchestra.run.smithi033.stdout:-------------------------------------------------------------------------------- 2024-04-15T21:35:37.367 INFO:teuthology.orchestra.run.smithi033.stdout:Total 43 MB/s | 77 MB 00:01 2024-04-15T21:35:37.367 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction check 2024-04-15T21:35:37.820 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction check succeeded. 2024-04-15T21:35:37.820 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction test 2024-04-15T21:35:38.685 INFO:teuthology.orchestra.run.smithi033.stdout:Transaction test succeeded. 2024-04-15T21:35:38.685 INFO:teuthology.orchestra.run.smithi033.stdout:Running transaction 2024-04-15T21:35:39.839 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T21:35:39.973 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T21:35:41.134 INFO:teuthology.orchestra.run.smithi033.stdout: Preparing : 1/1 2024-04-15T21:35:41.758 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-core-5.14.0-438.el9.x86_64 1/4 2024-04-15T21:35:41.855 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T21:35:44.737 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T21:35:44.875 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T21:35:45.044 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T21:35:45.089 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T21:35:49.953 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T21:35:49.991 INFO:teuthology.orchestra.run.smithi033.stdout: Installing : kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T21:35:50.185 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T21:35:55.085 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T21:36:21.158 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T21:36:21.158 INFO:teuthology.orchestra.run.smithi007.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T21:36:21.159 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:36:21.204 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T21:36:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T21:36:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T21:36:21.609 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T21:36:21.610 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T21:36:22.004 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T21:36:22.004 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:36:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-04-15T21:36:22.004 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T21:36:22.004 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T21:36:22.004 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T21:36:22.004 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T21:36:22.004 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:36:22.004 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-04-15T21:36:22.330 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T21:36:22.446 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T21:36:22.446 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T21:36:22.446 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T21:36:22.446 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T21:36:22.446 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T21:36:22.447 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-15T21:36:22.447 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T21:36:22.447 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T21:36:22.529 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T21:36:22.529 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T21:36:22.530 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2024-04-15T21:36:22.567 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2024-04-15T21:36:22.567 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2024-04-15T21:36:22.567 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.06 2024-04-15T21:36:22.567 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 68.el9 2024-04-15T21:36:22.567 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2024-04-15T21:36:22.567 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:Group : Unspecified 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 8264254 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : builder@centos.org 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : CentOS 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T21:36:22.568 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T21:36:22.569 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T21:36:22.569 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2024-04-15T21:36:22.569 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:36:22.569 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T21:36:22.570 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T21:36:22.570 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 5.14.0-438.el9.x86_64 2024-04-15T21:36:22.570 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T21:36:23.191 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2024-04-15T21:36:24.618 INFO:teuthology.orchestra.run.smithi007.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T21:36:24.649 INFO:teuthology.orchestra.run.smithi007.stderr:done 2024-04-15T21:36:24.652 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-04-15T21:36:24.667 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.3ZqaoFyAwl 2024-04-15T21:36:24.667 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.3ZqaoFyAwl 2024-04-15T21:36:24.740 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.3ZqaoFyAwl 2024-04-15T21:36:24.882 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.3ZqaoFyAwl is 6KB 2024-04-15T21:36:24.902 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.3ZqaoFyAwl 2024-04-15T21:36:24.916 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T21:36:24.985 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T21:36:24.985 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T21:36:24.985 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T21:36:24.985 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T21:36:24.986 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T21:36:25.155 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2024-04-15T21:36:25.923 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T21:36:25.923 INFO:teuthology.orchestra.run.smithi033.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-15T21:36:25.924 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-15T21:36:25.968 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-modules-5.14.0-438.el9.x86_64 4/4 2024-04-15T21:36:26.345 INFO:teuthology.orchestra.run.smithi033.stdout: Running scriptlet: kernel-5.14.0-438.el9.x86_64 4/4 2024-04-15T21:36:26.345 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-5.14.0-438.el9.x86_64 1/4 2024-04-15T21:36:26.345 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-core-5.14.0-438.el9.x86_64 2/4 2024-04-15T21:36:26.345 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-5.14.0-438.el9.x86_64 3/4 2024-04-15T21:36:26.776 INFO:teuthology.orchestra.run.smithi033.stdout: Verifying : kernel-modules-core-5.14.0-438.el9.x86_64 4/4 2024-04-15T21:36:26.776 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-15T21:36:26.776 INFO:teuthology.orchestra.run.smithi033.stdout:Installed: 2024-04-15T21:36:26.776 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-5.14.0-438.el9.x86_64 2024-04-15T21:36:26.776 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-core-5.14.0-438.el9.x86_64 2024-04-15T21:36:26.776 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-5.14.0-438.el9.x86_64 2024-04-15T21:36:26.777 INFO:teuthology.orchestra.run.smithi033.stdout: kernel-modules-core-5.14.0-438.el9.x86_64 2024-04-15T21:36:26.777 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-15T21:36:26.777 INFO:teuthology.orchestra.run.smithi033.stdout:Complete! 2024-04-15T21:36:27.128 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T21:36:27.225 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T21:36:27.225 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T21:36:27.225 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-438.el9.x86_64 2024-04-15T21:36:27.225 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-15T21:36:27.225 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-15T21:36:27.226 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-04-15T21:36:27.226 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi033.front.sepia.ceph.com, path=None, version=distro) 2024-04-15T21:36:27.226 DEBUG:teuthology.orchestra.run.smithi033:> rpm -q kernel | sort -rV | head -n 1 2024-04-15T21:36:27.308 INFO:teuthology.orchestra.run.smithi033.stdout:kernel-5.14.0-438.el9.x86_64 2024-04-15T21:36:27.309 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-438.el9.x86_64 2024-04-15T21:36:27.309 DEBUG:teuthology.orchestra.run.smithi033:> sudo rpm -qi grub2-tools 2024-04-15T21:36:27.346 INFO:teuthology.orchestra.run.smithi033.stdout:Name : grub2-tools 2024-04-15T21:36:27.347 INFO:teuthology.orchestra.run.smithi033.stdout:Epoch : 1 2024-04-15T21:36:27.347 INFO:teuthology.orchestra.run.smithi033.stdout:Version : 2.06 2024-04-15T21:36:27.347 INFO:teuthology.orchestra.run.smithi033.stdout:Release : 68.el9 2024-04-15T21:36:27.347 INFO:teuthology.orchestra.run.smithi033.stdout:Architecture: x86_64 2024-04-15T21:36:27.347 INFO:teuthology.orchestra.run.smithi033.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-15T21:36:27.347 INFO:teuthology.orchestra.run.smithi033.stdout:Group : Unspecified 2024-04-15T21:36:27.347 INFO:teuthology.orchestra.run.smithi033.stdout:Size : 8264254 2024-04-15T21:36:27.347 INFO:teuthology.orchestra.run.smithi033.stdout:License : GPLv3+ 2024-04-15T21:36:27.347 INFO:teuthology.orchestra.run.smithi033.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-15T21:36:27.347 INFO:teuthology.orchestra.run.smithi033.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-15T21:36:27.347 INFO:teuthology.orchestra.run.smithi033.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-15T21:36:27.348 INFO:teuthology.orchestra.run.smithi033.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-15T21:36:27.348 INFO:teuthology.orchestra.run.smithi033.stdout:Packager : builder@centos.org 2024-04-15T21:36:27.348 INFO:teuthology.orchestra.run.smithi033.stdout:Vendor : CentOS 2024-04-15T21:36:27.348 INFO:teuthology.orchestra.run.smithi033.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-15T21:36:27.348 INFO:teuthology.orchestra.run.smithi033.stdout:Summary : Support tools for GRUB. 2024-04-15T21:36:27.348 INFO:teuthology.orchestra.run.smithi033.stdout:Description : 2024-04-15T21:36:27.348 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-15T21:36:27.348 INFO:teuthology.orchestra.run.smithi033.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-15T21:36:27.348 INFO:teuthology.orchestra.run.smithi033.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-15T21:36:27.348 INFO:teuthology.orchestra.run.smithi033.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-15T21:36:27.348 INFO:teuthology.orchestra.run.smithi033.stdout:hardware devices. 2024-04-15T21:36:27.348 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-15T21:36:27.348 INFO:teuthology.orchestra.run.smithi033.stdout:This subpackage provides tools for support of all platforms. 2024-04-15T21:36:27.349 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-15T21:36:27.350 INFO:teuthology.task.kernel:Updating grub on smithi033 to boot 5.14.0-438.el9.x86_64 2024-04-15T21:36:27.350 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-15T21:36:27.929 INFO:teuthology.orchestra.run.smithi033.stderr:Generating grub configuration file ... 2024-04-15T21:36:29.328 INFO:teuthology.orchestra.run.smithi033.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-15T21:36:29.364 INFO:teuthology.orchestra.run.smithi033.stderr:done 2024-04-15T21:36:29.366 DEBUG:teuthology.orchestra.run.smithi033:> mktemp 2024-04-15T21:36:29.381 INFO:teuthology.orchestra.run.smithi033.stdout:/tmp/tmp.zWHC25LDwx 2024-04-15T21:36:29.381 DEBUG:teuthology.orchestra.run.smithi033:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zWHC25LDwx 2024-04-15T21:36:29.454 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0666 /tmp/tmp.zWHC25LDwx 2024-04-15T21:36:29.599 DEBUG:teuthology.orchestra.remote:smithi033:/tmp/tmp.zWHC25LDwx is 6KB 2024-04-15T21:36:29.647 DEBUG:teuthology.orchestra.run.smithi033:> rm -fr /tmp/tmp.zWHC25LDwx 2024-04-15T21:36:29.662 DEBUG:teuthology.orchestra.run.smithi033:> sudo /bin/ls /boot/loader/entries || true 2024-04-15T21:36:29.730 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-15T21:36:29.731 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-15T21:36:29.731 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-15T21:36:29.731 INFO:teuthology.orchestra.run.smithi033.stdout:15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64.conf 2024-04-15T21:36:29.732 DEBUG:teuthology.orchestra.run.smithi033:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-438.el9.x86_64 2024-04-15T21:36:29.863 DEBUG:teuthology.orchestra.run.smithi033:> sudo shutdown -r now 2024-04-15T21:36:55.160 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-15T21:36:55.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-04-15T21:36:55.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T21:36:59.866 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-04-15T21:36:59.867 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-15T21:36:59.868 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T21:37:13.506 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2024-04-15T21:37:18.366 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-04-15T21:37:22.510 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-04-15T21:37:22.511 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T21:37:25.566 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2024-04-15T21:37:27.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-15T21:37:27.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T21:37:37.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-04-15T21:37:37.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T21:37:45.762 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.33 2024-04-15T21:37:57.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-15T21:37:57.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T21:37:59.155 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-04-15T21:37:59.581 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-15T21:37:59.581 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-438.el9.x86_64"... 2024-04-15T21:37:59.582 DEBUG:teuthology.orchestra.run.smithi033:> uname -r 2024-04-15T21:37:59.598 INFO:teuthology.orchestra.run.smithi033.stdout:5.14.0-438.el9.x86_64 2024-04-15T21:37:59.598 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T21:37:59.599 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T21:37:59.599 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T21:38:00.599 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-04-15T21:38:00.599 DEBUG:teuthology.orchestra.run.smithi033:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T21:38:00.702 INFO:teuthology.orchestra.run.smithi033.stdout:ttyS1 2024-04-15T21:38:00.756 DEBUG:teuthology.parallel:result is None 2024-04-15T21:38:09.643 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-04-15T21:38:10.320 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-04-15T21:38:10.321 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-438.el9.x86_64"... 2024-04-15T21:38:10.321 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-04-15T21:38:10.335 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-438.el9.x86_64 2024-04-15T21:38:10.335 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-438.el9.x86_64 vs 5.14.0-438.el9.x86_64 2024-04-15T21:38:10.336 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-15T21:38:10.336 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-15T21:38:11.337 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-15T21:38:11.337 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-15T21:38:11.389 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2024-04-15T21:38:11.423 DEBUG:teuthology.parallel:result is None 2024-04-15T21:38:11.424 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-15T21:38:11.431 INFO:teuthology.task.internal:Creating test directory... 2024-04-15T21:38:11.431 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T21:38:11.433 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-15T21:38:11.454 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-15T21:38:11.460 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-15T21:38:11.470 INFO:teuthology.task.internal:Creating archive directory... 2024-04-15T21:38:11.470 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T21:38:11.491 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-15T21:38:11.532 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-15T21:38:11.539 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-15T21:38:11.539 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T21:38:11.569 DEBUG:teuthology.orchestra.run.smithi033:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-15T21:38:11.605 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T21:38:11.606 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T21:38:11.620 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T21:38:11.621 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-15T21:38:11.622 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-15T21:38:11.629 INFO:teuthology.task.internal:Configuring sudo... 2024-04-15T21:38:11.629 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T21:38:11.664 DEBUG:teuthology.orchestra.run.smithi033:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-15T21:38:11.696 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-15T21:38:11.705 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-15T21:38:11.705 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T21:38:11.735 DEBUG:teuthology.orchestra.run.smithi033:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-15T21:38:11.753 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T21:38:11.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T21:38:11.886 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T21:38:11.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T21:38:12.013 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-15T21:38:12.013 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T21:38:12.082 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T21:38:12.127 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-15T21:38:12.181 DEBUG:teuthology.orchestra.run.smithi033:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T21:38:12.263 DEBUG:teuthology.orchestra.run.smithi033:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-15T21:38:12.311 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-15T21:38:12.311 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-15T21:38:12.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-04-15T21:38:12.385 DEBUG:teuthology.orchestra.run.smithi033:> sudo service rsyslog restart 2024-04-15T21:38:12.429 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T21:38:12.480 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T21:38:12.946 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-15T21:38:12.954 INFO:teuthology.task.internal:Starting timer... 2024-04-15T21:38:12.954 INFO:teuthology.run_tasks:Running task pcp... 2024-04-15T21:38:12.965 INFO:teuthology.run_tasks:Running task selinux... 2024-04-15T21:38:12.974 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2024-04-15T21:38:13.024 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2024-04-15T21:38:13.026 DEBUG:teuthology.orchestra.run.smithi033:> sudo service auditd rotate 2024-04-15T21:38:13.088 INFO:teuthology.orchestra.run.smithi033.stdout:Rotating logs: 2024-04-15T21:38:13.091 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-15T21:38:13.091 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2024-04-15T21:38:13.135 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2024-04-15T21:38:13.135 DEBUG:teuthology.orchestra.run.smithi033:> /usr/sbin/getenforce 2024-04-15T21:38:13.157 INFO:teuthology.orchestra.run.smithi033.stdout:Permissive 2024-04-15T21:38:13.158 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi033.front.sepia.ceph.com': 'permissive'} 2024-04-15T21:38:13.158 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-15T21:38:13.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T21:38:13.189 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-15T21:38:13.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T21:38:13.229 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-15T21:38:13.229 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2024-04-15T21:38:13.278 DEBUG:teuthology.orchestra.run.smithi033:> sudo /usr/sbin/setenforce permissive 2024-04-15T21:38:13.310 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-15T21:38:13.319 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-15T21:38:13.337 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-15T21:38:13.338 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi033.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-15T21:42:45.607 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi033.front.sepia.ceph.com')] 2024-04-15T21:42:45.609 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-04-15T21:42:45.610 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T21:42:45.708 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-04-15T21:42:45.790 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-04-15T21:42:45.791 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-15T21:42:45.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi033.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-15T21:42:45.884 DEBUG:teuthology.orchestra.run.smithi033:> true 2024-04-15T21:42:45.967 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi033.front.sepia.ceph.com' 2024-04-15T21:42:45.968 INFO:teuthology.run_tasks:Running task clock... 2024-04-15T21:42:45.977 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-15T21:42:45.978 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T21:42:45.978 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T21:42:45.981 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-15T21:42:45.981 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T21:42:46.015 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T21:42:46.034 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T21:42:46.045 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-15T21:42:46.065 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-15T21:42:46.067 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2024-04-15T21:42:46.082 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2024-04-15T21:42:46.098 INFO:teuthology.orchestra.run.smithi033.stderr:sudo: ntpd: command not found 2024-04-15T21:42:46.100 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T21:42:46.113 INFO:teuthology.orchestra.run.smithi033.stdout:506 Cannot talk to daemon 2024-04-15T21:42:46.117 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T21:42:46.131 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-15T21:42:46.149 INFO:teuthology.orchestra.run.smithi033.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-15T21:42:46.171 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-04-15T21:42:46.176 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T21:42:46.176 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-04-15T21:42:46.176 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T21:42:46.176 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T21:42:46.176 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T21:42:46.176 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T21:42:46.201 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-04-15T21:42:46.205 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T21:42:46.205 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-04-15T21:42:46.205 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T21:42:46.206 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T21:42:46.206 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T21:42:46.206 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T21:42:46.206 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-15T21:42:46.314 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'conf': {'global': {'mon election default strategy': 3}, 'mgr': {'debug mgr': 20, 'debug ms': 1, 'mgr/cephadm/use_agent': False}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'a9a752df26c63acad72e1b3569fd79a515ca0765'} 2024-04-15T21:42:46.314 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-15T21:42:46.315 INFO:tasks.cephadm:Cluster fsid is 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:42:46.315 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-15T21:42:46.315 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.7', 'mon.c': '[v2:172.21.15.7:3301,v1:172.21.15.7:6790]', 'mon.b': '172.21.15.33'} 2024-04-15T21:42:46.315 INFO:tasks.cephadm:First mon is mon.a on smithi007 2024-04-15T21:42:46.315 INFO:tasks.cephadm:First mgr is y 2024-04-15T21:42:46.315 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-15T21:42:46.315 DEBUG:teuthology.orchestra.run.smithi007:> sudo hostname $(hostname -s) 2024-04-15T21:42:46.345 DEBUG:teuthology.orchestra.run.smithi033:> sudo hostname $(hostname -s) 2024-04-15T21:42:46.374 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-15T21:42:46.374 DEBUG:teuthology.orchestra.run.smithi007:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-15T21:42:46.675 INFO:teuthology.orchestra.run.smithi007.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 15 21:42 /home/ubuntu/cephtest/cephadm 2024-04-15T21:42:46.675 DEBUG:teuthology.orchestra.run.smithi033:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-15T21:42:46.841 INFO:teuthology.orchestra.run.smithi033.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 15 21:42 /home/ubuntu/cephtest/cephadm 2024-04-15T21:42:46.842 DEBUG:teuthology.orchestra.run.smithi007:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-15T21:42:46.859 DEBUG:teuthology.orchestra.run.smithi033:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-15T21:42:46.898 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-15T21:42:46.898 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-15T21:42:46.910 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-15T21:42:47.748 INFO:teuthology.orchestra.run.smithi007.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-15T21:42:47.758 INFO:teuthology.orchestra.run.smithi033.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-15T21:43:10.833 INFO:teuthology.orchestra.run.smithi033.stdout:{ 2024-04-15T21:43:10.833 INFO:teuthology.orchestra.run.smithi033.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-15T21:43:10.834 INFO:teuthology.orchestra.run.smithi033.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-15T21:43:10.834 INFO:teuthology.orchestra.run.smithi033.stdout: "repo_digests": [ 2024-04-15T21:43:10.834 INFO:teuthology.orchestra.run.smithi033.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-15T21:43:10.834 INFO:teuthology.orchestra.run.smithi033.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-15T21:43:10.834 INFO:teuthology.orchestra.run.smithi033.stdout: ] 2024-04-15T21:43:10.834 INFO:teuthology.orchestra.run.smithi033.stdout:} 2024-04-15T21:43:11.018 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-15T21:43:11.018 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-15T21:43:11.018 INFO:teuthology.orchestra.run.smithi007.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-15T21:43:11.018 INFO:teuthology.orchestra.run.smithi007.stdout: "repo_digests": [ 2024-04-15T21:43:11.019 INFO:teuthology.orchestra.run.smithi007.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-15T21:43:11.019 INFO:teuthology.orchestra.run.smithi007.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-15T21:43:11.019 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-15T21:43:11.019 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-15T21:43:11.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph 2024-04-15T21:43:11.076 DEBUG:teuthology.orchestra.run.smithi033:> sudo mkdir -p /etc/ceph 2024-04-15T21:43:11.108 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /etc/ceph 2024-04-15T21:43:11.145 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 777 /etc/ceph 2024-04-15T21:43:11.178 INFO:tasks.cephadm:Writing seed config... 2024-04-15T21:43:11.179 INFO:tasks.cephadm: override: [global] mon election default strategy = 3 2024-04-15T21:43:11.179 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-15T21:43:11.180 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-15T21:43:11.180 INFO:tasks.cephadm: override: [mgr] mgr/cephadm/use_agent = False 2024-04-15T21:43:11.180 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-15T21:43:11.180 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-15T21:43:11.180 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-15T21:43:11.180 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-15T21:43:11.180 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-15T21:43:11.181 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-15T21:43:11.181 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-15T21:43:11.203 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 18ed60a0-fb71-11ee-bc8f-c7b262605968 mon election default strategy = 3 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 mgr/cephadm/use_agent = False [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-15T21:43:11.203 DEBUG:teuthology.orchestra.run.smithi007:mon.a> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.a.service 2024-04-15T21:43:11.247 DEBUG:teuthology.orchestra.run.smithi007:mgr.y> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mgr.y.service 2024-04-15T21:43:11.289 INFO:tasks.cephadm:Bootstrapping... 2024-04-15T21:43:11.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.7 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-15T21:43:11.492 INFO:teuthology.orchestra.run.smithi007.stderr:-------------------------------------------------------------------------------- 2024-04-15T21:43:11.492 INFO:teuthology.orchestra.run.smithi007.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '18ed60a0-fb71-11ee-bc8f-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.7', '--skip-admin-label'] 2024-04-15T21:43:11.515 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: 4.9.4-dev 2024-04-15T21:43:11.517 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying podman|docker is present... 2024-04-15T21:43:11.538 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: 4.9.4-dev 2024-04-15T21:43:11.540 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying lvm2 is present... 2024-04-15T21:43:11.540 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying time synchronization is in place... 2024-04-15T21:43:11.547 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-15T21:43:11.554 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2024-04-15T21:43:11.560 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: enabled 2024-04-15T21:43:11.567 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: active 2024-04-15T21:43:11.567 INFO:teuthology.orchestra.run.smithi007.stderr:Unit chronyd.service is enabled and running 2024-04-15T21:43:11.567 INFO:teuthology.orchestra.run.smithi007.stderr:Repeating the final host check... 2024-04-15T21:43:11.588 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: 4.9.4-dev 2024-04-15T21:43:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-15T21:43:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl is present 2024-04-15T21:43:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:lvcreate is present 2024-04-15T21:43:11.596 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-15T21:43:11.602 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2024-04-15T21:43:11.609 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: enabled 2024-04-15T21:43:11.616 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: active 2024-04-15T21:43:11.616 INFO:teuthology.orchestra.run.smithi007.stderr:Unit chronyd.service is enabled and running 2024-04-15T21:43:11.617 INFO:teuthology.orchestra.run.smithi007.stderr:Host looks OK 2024-04-15T21:43:11.617 INFO:teuthology.orchestra.run.smithi007.stderr:Cluster fsid: 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:43:11.617 INFO:teuthology.orchestra.run.smithi007.stderr:Acquiring lock 140522240851872 on /run/cephadm/18ed60a0-fb71-11ee-bc8f-c7b262605968.lock 2024-04-15T21:43:11.617 INFO:teuthology.orchestra.run.smithi007.stderr:Lock 140522240851872 acquired on /run/cephadm/18ed60a0-fb71-11ee-bc8f-c7b262605968.lock 2024-04-15T21:43:11.617 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying IP 172.21.15.7 port 3300 ... 2024-04-15T21:43:11.617 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying IP 172.21.15.7 port 6789 ... 2024-04-15T21:43:11.618 INFO:teuthology.orchestra.run.smithi007.stderr:Base mon IP is 172.21.15.7, final addrv is [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-04-15T21:43:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.7 metric 100 2024-04-15T21:43:11.620 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.7 metric 100 2024-04-15T21:43:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-15T21:43:11.622 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-15T21:43:11.623 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-15T21:43:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-15T21:43:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-15T21:43:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-15T21:43:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-15T21:43:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:15ea/64 scope link noprefixroute 2024-04-15T21:43:11.625 INFO:teuthology.orchestra.run.smithi007.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-15T21:43:11.626 INFO:teuthology.orchestra.run.smithi007.stderr:Mon IP `172.21.15.7` is in CIDR network `172.21.0.0/20` 2024-04-15T21:43:11.626 INFO:teuthology.orchestra.run.smithi007.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-15T21:43:11.627 INFO:teuthology.orchestra.run.smithi007.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-15T21:43:11.730 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-15T21:43:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Getting image source signatures 2024-04-15T21:43:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-15T21:43:12.256 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-15T21:43:12.257 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-15T21:43:12.257 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-15T21:43:12.257 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-15T21:43:12.257 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-15T21:43:12.397 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: Writing manifest to image destination 2024-04-15T21:43:12.550 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-15T21:43:12.950 INFO:teuthology.orchestra.run.smithi007.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-15T21:43:13.516 INFO:teuthology.orchestra.run.smithi007.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-15T21:43:13.516 INFO:teuthology.orchestra.run.smithi007.stderr:Extracting ceph user uid/gid from container image... 2024-04-15T21:43:13.941 INFO:teuthology.orchestra.run.smithi007.stderr:stat: 167 167 2024-04-15T21:43:14.438 INFO:teuthology.orchestra.run.smithi007.stderr:Creating initial keys... 2024-04-15T21:43:14.955 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-authtool: AQBynx1mSLlkLBAACnRoxZTHXPXoChFZFjRnmA== 2024-04-15T21:43:15.752 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-authtool: AQBznx1mUQxNJxAApCLMb4IQEZGXnw9Nrsn2eQ== 2024-04-15T21:43:16.658 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph-authtool: AQB0nx1mcGmeIRAA7o85AAvf/W830+8xojZi8A== 2024-04-15T21:43:17.057 INFO:teuthology.orchestra.run.smithi007.stderr:Creating initial monmap... 2024-04-15T21:43:17.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-15T21:43:17.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-15T21:43:17.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:43:17.472 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-15T21:43:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:monmaptool for a [v2:172.21.15.7:3300,v1:172.21.15.7:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-15T21:43:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:setting min_mon_release = octopus 2024-04-15T21:43:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: set fsid to 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:43:17.936 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-15T21:43:17.936 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-15T21:43:17.937 INFO:teuthology.orchestra.run.smithi007.stderr:Creating mon... 2024-04-15T21:43:18.808 INFO:teuthology.orchestra.run.smithi007.stderr:create mon.a on 2024-04-15T21:43:19.011 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-15T21:43:19.173 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968.target. 2024-04-15T21:43:19.173 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968.target → /etc/systemd/system/ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968.target. 2024-04-15T21:43:19.481 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to reset failed state of unit ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.a.service: Unit ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.a.service not loaded. 2024-04-15T21:43:19.489 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968.target.wants/ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.a.service → /etc/systemd/system/ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@.service. 2024-04-15T21:43:20.185 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:20 smithi007 ceph-mon[24482]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-15T21:43:20.186 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:20 smithi007 bash[24462]: 29a6848daebaa6bbbceeafd18b14f279cebf8cf456a04a635fe160b39087474e 2024-04-15T21:43:20.199 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: disabled 2024-04-15T21:43:20.206 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2024-04-15T21:43:20.207 INFO:teuthology.orchestra.run.smithi007.stderr:firewalld.service is not enabled 2024-04-15T21:43:20.207 INFO:teuthology.orchestra.run.smithi007.stderr:Not possible to enable service . firewalld.service is not available 2024-04-15T21:43:20.207 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mon to start... 2024-04-15T21:43:20.207 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mon... 2024-04-15T21:43:20.440 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:20 smithi007 systemd[1]: Started Ceph mon.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:43:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: cluster: 2024-04-15T21:43:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: id: 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:43:20.719 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-15T21:43:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-15T21:43:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: services: 2024-04-15T21:43:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.593551s) 2024-04-15T21:43:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-15T21:43:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-15T21:43:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-15T21:43:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: data: 2024-04-15T21:43:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-15T21:43:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-15T21:43:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-15T21:43:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: pgs: 2024-04-15T21:43:20.720 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-15T21:43:21.219 INFO:teuthology.orchestra.run.smithi007.stderr:mon is available 2024-04-15T21:43:21.220 INFO:teuthology.orchestra.run.smithi007.stderr:Assimilating anything we can from ceph.conf... 2024-04-15T21:43:21.432 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:21 smithi007 ceph-mon[24482]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-15T21:43:21.432 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:21 smithi007 ceph-mon[24482]: monmap e1: 1 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]} 2024-04-15T21:43:21.432 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:21 smithi007 ceph-mon[24482]: fsmap 2024-04-15T21:43:21.432 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:21 smithi007 ceph-mon[24482]: osdmap e1: 0 total, 0 up, 0 in 2024-04-15T21:43:21.432 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:21 smithi007 ceph-mon[24482]: mgrmap e1: no daemons active 2024-04-15T21:43:21.432 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:21 smithi007 ceph-mon[24482]: from='client.? 172.21.15.7:0/4174154492' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-15T21:43:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-15T21:43:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [global] 2024-04-15T21:43:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: fsid = 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:43:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.7:3300,v1:172.21.15.7:6789] 2024-04-15T21:43:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-15T21:43:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-15T21:43:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-15T21:43:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-15T21:43:21.725 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-15T21:43:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [mgr] 2024-04-15T21:43:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mgr/cephadm/use_agent = False 2024-04-15T21:43:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-15T21:43:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-15T21:43:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [osd] 2024-04-15T21:43:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-15T21:43:21.726 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-15T21:43:22.242 INFO:teuthology.orchestra.run.smithi007.stderr:Generating new minimal ceph.conf... 2024-04-15T21:43:22.463 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:22 smithi007 ceph-mon[24482]: from='client.? 172.21.15.7:0/2053263992' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-15T21:43:22.463 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:22 smithi007 ceph-mon[24482]: from='client.? 172.21.15.7:0/2053263992' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-15T21:43:23.215 INFO:teuthology.orchestra.run.smithi007.stderr:Restarting the monitor... 2024-04-15T21:43:23.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:23 smithi007 ceph-mon[24482]: from='client.? 172.21.15.7:0/3417436042' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:43:23.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:23 smithi007 systemd[1]: Stopping Ceph mon.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:43:23.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:23 smithi007 bash[24700]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon.a" found: no such container 2024-04-15T21:43:23.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:23 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a[24478]: 2024-04-15T21:43:23.371+0000 7f1f8d16c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T21:43:23.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:23 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a[24478]: 2024-04-15T21:43:23.371+0000 7f1f8d16c700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-15T21:43:23.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:23 smithi007 podman[24713]: 2024-04-15 21:43:23.393246632 +0000 UTC m=+0.121759530 container died 29a6848daebaa6bbbceeafd18b14f279cebf8cf456a04a635fe160b39087474e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a, GIT_CLEAN=True, architecture=x86_64, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, ceph=True, io.openshift.expose-services=, name=centos-stream, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-15T21:43:23.673 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:23 smithi007 podman[24713]: 2024-04-15 21:43:23.671513991 +0000 UTC m=+0.400026896 container cleanup 29a6848daebaa6bbbceeafd18b14f279cebf8cf456a04a635fe160b39087474e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a, com.redhat.component=centos-stream-container, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, io.openshift.expose-services=, maintainer=Guillaume Abrioux , architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754) 2024-04-15T21:43:24.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:23 smithi007 bash[24713]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a 2024-04-15T21:43:24.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:23 smithi007 bash[24743]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon.a" found: no such container 2024-04-15T21:43:24.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:23 smithi007 podman[24727]: 2024-04-15 21:43:23.889168028 +0000 UTC m=+0.495475588 container remove 29a6848daebaa6bbbceeafd18b14f279cebf8cf456a04a635fe160b39087474e (image=quay.io/ceph/ceph:v17.2.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a, io.openshift.expose-services=, RELEASE=HEAD, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, ceph=True, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-15T21:43:24.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:23 smithi007 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-15T21:43:24.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:23 smithi007 systemd[1]: Stopped Ceph mon.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:43:24.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:23 smithi007 systemd[1]: Starting Ceph mon.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:43:24.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 podman[24807]: 2024-04-15 21:43:24.01902605 +0000 UTC m=+0.019169123 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-15T21:43:24.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 podman[24807]: 2024-04-15 21:43:24.139971997 +0000 UTC m=+0.140115044 container create f3463cf07a9237a4fbc6c53496166800dcfcb18e04896afe06467374e4e64f3f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vendor=Red Hat, Inc., name=centos-stream, io.openshift.expose-services=, architecture=x86_64, com.redhat.component=centos-stream-container, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, release=754, version=8, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD) 2024-04-15T21:43:24.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 podman[24807]: 2024-04-15 21:43:24.30895958 +0000 UTC m=+0.309102629 container init f3463cf07a9237a4fbc6c53496166800dcfcb18e04896afe06467374e4e64f3f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , version=8, vcs-type=git, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, architecture=x86_64, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, vendor=Red Hat, Inc., release=754, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, RELEASE=HEAD) 2024-04-15T21:43:24.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 podman[24807]: 2024-04-15 21:43:24.312167658 +0000 UTC m=+0.312310705 container start f3463cf07a9237a4fbc6c53496166800dcfcb18e04896afe06467374e4e64f3f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, release=754, maintainer=Guillaume Abrioux , architecture=x86_64, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-15T21:43:24.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: set uid:gid to 167:167 (ceph:ceph) 2024-04-15T21:43:24.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-15T21:43:24.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: pidfile_write: ignore empty --pid-file 2024-04-15T21:43:24.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: load: jerasure load: lrc 2024-04-15T21:43:24.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: RocksDB version: 6.15.5 2024-04-15T21:43:24.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-15T21:43:24.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Compile date Apr 18 2022 2024-04-15T21:43:24.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: DB SUMMARY 2024-04-15T21:43:24.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: DB Session ID: NCCJG7RC9IWZUML942IH 2024-04-15T21:43:24.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: CURRENT file: CURRENT 2024-04-15T21:43:24.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: IDENTITY file: IDENTITY 2024-04-15T21:43:24.395 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-15T21:43:24.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-15T21:43:24.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 82609 ; 2024-04-15T21:43:24.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.error_if_exists: 0 2024-04-15T21:43:24.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.create_if_missing: 0 2024-04-15T21:43:24.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.paranoid_checks: 1 2024-04-15T21:43:24.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-15T21:43:24.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.env: 0x55eef3ec6860 2024-04-15T21:43:24.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.fs: Posix File System 2024-04-15T21:43:24.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.info_log: 0x55eef5fd7dc0 2024-04-15T21:43:24.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_file_opening_threads: 16 2024-04-15T21:43:24.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.statistics: (nil) 2024-04-15T21:43:24.396 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.use_fsync: 0 2024-04-15T21:43:24.397 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_log_file_size: 0 2024-04-15T21:43:24.397 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-15T21:43:24.398 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-15T21:43:24.398 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.keep_log_file_num: 1000 2024-04-15T21:43:24.398 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.recycle_log_file_num: 0 2024-04-15T21:43:24.398 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.allow_fallocate: 1 2024-04-15T21:43:24.398 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.allow_mmap_reads: 0 2024-04-15T21:43:24.399 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.allow_mmap_writes: 0 2024-04-15T21:43:24.399 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.use_direct_reads: 0 2024-04-15T21:43:24.399 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-15T21:43:24.399 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.create_missing_column_families: 0 2024-04-15T21:43:24.399 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.db_log_dir: 2024-04-15T21:43:24.399 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-04-15T21:43:24.399 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-15T21:43:24.399 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-15T21:43:24.399 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-15T21:43:24.399 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-15T21:43:24.399 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-15T21:43:24.399 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.advise_random_on_open: 1 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.db_write_buffer_size: 0 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.write_buffer_manager: 0x55eef60c82a0 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.rate_limiter: (nil) 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.wal_recovery_mode: 2 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.enable_thread_tracking: 0 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.enable_pipelined_write: 0 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.unordered_write: 0 2024-04-15T21:43:24.400 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-15T21:43:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-15T21:43:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-15T21:43:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-15T21:43:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.row_cache: None 2024-04-15T21:43:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.wal_filter: None 2024-04-15T21:43:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-15T21:43:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.allow_ingest_behind: 0 2024-04-15T21:43:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.preserve_deletes: 0 2024-04-15T21:43:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.two_write_queues: 0 2024-04-15T21:43:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.manual_wal_flush: 0 2024-04-15T21:43:24.401 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.atomic_flush: 0 2024-04-15T21:43:24.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-15T21:43:24.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-15T21:43:24.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-15T21:43:24.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.log_readahead_size: 0 2024-04-15T21:43:24.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-15T21:43:24.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.best_efforts_recovery: 0 2024-04-15T21:43:24.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-15T21:43:24.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-15T21:43:24.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.allow_data_in_errors: 0 2024-04-15T21:43:24.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.db_host_id: __hostname__ 2024-04-15T21:43:24.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_background_jobs: 2 2024-04-15T21:43:24.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_background_compactions: -1 2024-04-15T21:43:24.403 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_subcompactions: 1 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_total_wal_size: 0 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_open_files: -1 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.bytes_per_sync: 0 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compaction_readahead_size: 0 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_background_flushes: -1 2024-04-15T21:43:24.404 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Compression algorithms supported: 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: kZSTD supported: 0 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: kXpressCompression supported: 0 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: kLZ4HCCompression supported: 1 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: kLZ4Compression supported: 1 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: kBZip2Compression supported: 0 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: kZlibCompression supported: 1 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: kSnappyCompression supported: 1 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.merge_operator: 2024-04-15T21:43:24.405 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compaction_filter: None 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compaction_filter_factory: None 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.sst_partitioner_factory: None 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eef5fa3d00) 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks: 1 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout: pin_top_level_index_and_filter: 1 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_type: 0 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_index_type: 0 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_shortening: 1 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-15T21:43:24.406 INFO:journalctl@ceph.mon.a.smithi007.stdout: hash_index_allow_collision: 1 2024-04-15T21:43:24.407 INFO:journalctl@ceph.mon.a.smithi007.stdout: checksum: 1 2024-04-15T21:43:24.407 INFO:journalctl@ceph.mon.a.smithi007.stdout: no_block_cache: 0 2024-04-15T21:43:24.407 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache: 0x55eef600e170 2024-04-15T21:43:24.407 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_name: BinnedLRUCache 2024-04-15T21:43:24.407 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_options: 2024-04-15T21:43:24.407 INFO:journalctl@ceph.mon.a.smithi007.stdout: capacity : 536870912 2024-04-15T21:43:24.407 INFO:journalctl@ceph.mon.a.smithi007.stdout: num_shard_bits : 4 2024-04-15T21:43:24.407 INFO:journalctl@ceph.mon.a.smithi007.stdout: strict_capacity_limit : 0 2024-04-15T21:43:24.408 INFO:journalctl@ceph.mon.a.smithi007.stdout: high_pri_pool_ratio: 0.000 2024-04-15T21:43:24.408 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_cache_compressed: (nil) 2024-04-15T21:43:24.408 INFO:journalctl@ceph.mon.a.smithi007.stdout: persistent_cache: (nil) 2024-04-15T21:43:24.408 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size: 4096 2024-04-15T21:43:24.408 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_size_deviation: 10 2024-04-15T21:43:24.408 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_restart_interval: 16 2024-04-15T21:43:24.408 INFO:journalctl@ceph.mon.a.smithi007.stdout: index_block_restart_interval: 1 2024-04-15T21:43:24.408 INFO:journalctl@ceph.mon.a.smithi007.stdout: metadata_block_size: 4096 2024-04-15T21:43:24.408 INFO:journalctl@ceph.mon.a.smithi007.stdout: partition_filters: 0 2024-04-15T21:43:24.408 INFO:journalctl@ceph.mon.a.smithi007.stdout: use_delta_encoding: 1 2024-04-15T21:43:24.409 INFO:journalctl@ceph.mon.a.smithi007.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-15T21:43:24.409 INFO:journalctl@ceph.mon.a.smithi007.stdout: whole_key_filtering: 1 2024-04-15T21:43:24.409 INFO:journalctl@ceph.mon.a.smithi007.stdout: verify_compression: 0 2024-04-15T21:43:24.409 INFO:journalctl@ceph.mon.a.smithi007.stdout: read_amp_bytes_per_bit: 0 2024-04-15T21:43:24.409 INFO:journalctl@ceph.mon.a.smithi007.stdout: format_version: 4 2024-04-15T21:43:24.409 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_index_compression: 1 2024-04-15T21:43:24.409 INFO:journalctl@ceph.mon.a.smithi007.stdout: block_align: 0 2024-04-15T21:43:24.409 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.write_buffer_size: 33554432 2024-04-15T21:43:24.409 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_write_buffer_number: 2 2024-04-15T21:43:24.409 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compression: NoCompression 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.bottommost_compression: Disabled 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.prefix_extractor: nullptr 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.num_levels: 7 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-15T21:43:24.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compression_opts.level: 32767 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compression_opts.strategy: 0 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compression_opts.enabled: false 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.target_file_size_base: 67108864 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-15T21:43:24.411 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.arena_block_size: 4194304 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-15T21:43:24.412 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.disable_auto_compactions: 0 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.table_properties_collectors: 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.inplace_update_support: 0 2024-04-15T21:43:24.413 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.bloom_locality: 0 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.max_successive_merges: 0 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.paranoid_file_checks: 0 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.force_consistency_checks: 1 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.report_bg_io_stats: 0 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.ttl: 2592000 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.enable_blob_files: false 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.min_blob_size: 0 2024-04-15T21:43:24.414 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.blob_file_size: 268435456 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713217404338122, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713217404338848, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79159, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77422, "index_size": 225, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10027, "raw_average_key_size": 47, "raw_value_size": 71731, "raw_average_value_size": 339, "num_data_blocks": 10, "num_entries": 211, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713217404, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "cdc1a78e-be1b-4473-9c55-512f8c8275ed", "db_session_id": "NCCJG7RC9IWZUML942IH"}} 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713217404339177, "job": 1, "event": "recovery_finished"} 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-15T21:43:24.415 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55eef5ff4a80 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: DB pointer 0x55eef6004000 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** DB Stats ** 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-15T21:43:24.416 INFO:journalctl@ceph.mon.a.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: L0 2/0 79.06 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.7 0.00 0.00 1 0.000 0 0 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: Sum 2/0 79.06 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.7 0.00 0.00 1 0.000 0 0 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.7 0.00 0.00 1 0.000 0 0 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.7 0.00 0.00 1 0.000 0 0 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T21:43:24.417 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative compaction: 0.00 GB write, 25.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval compaction: 0.00 GB write, 25.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: L0 2/0 79.06 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.7 0.00 0.00 1 0.000 0 0 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: Sum 2/0 79.06 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 154.7 0.00 0.00 1 0.000 0 0 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-15T21:43:24.418 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-15T21:43:24.419 INFO:journalctl@ceph.mon.a.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T21:43:24.419 INFO:journalctl@ceph.mon.a.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T21:43:24.419 INFO:journalctl@ceph.mon.a.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 154.7 0.00 0.00 1 0.000 0 0 2024-04-15T21:43:24.419 INFO:journalctl@ceph.mon.a.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T21:43:24.419 INFO:journalctl@ceph.mon.a.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T21:43:24.419 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T21:43:24.419 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T21:43:24.419 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T21:43:24.419 INFO:journalctl@ceph.mon.a.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T21:43:24.419 INFO:journalctl@ceph.mon.a.smithi007.stdout: Cumulative compaction: 0.00 GB write, 24.78 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T21:43:24.419 INFO:journalctl@ceph.mon.a.smithi007.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T21:43:24.419 INFO:journalctl@ceph.mon.a.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T21:43:24.419 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: starting mon.a rank 0 at public addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] at bind addrs [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: mon.a@-1(???) e1 preinit fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: mon.a@-1(???).mds e1 new map 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: mon.a@-1(???).mds e1 print_map 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout: e1 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout: legacy client fscid: -1 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout: 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout: No filesystems configured 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T21:43:24.420 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: expand_channel_meta expand map: {default=false} 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: expand_channel_meta from 'false' to 'false' 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: expand_channel_meta expanded map: {default=false} 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: expand_channel_meta expand map: {default=info} 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: expand_channel_meta from 'info' to 'info' 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: expand_channel_meta expanded map: {default=info} 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: expand_channel_meta expand map: {default=daemon} 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: expand_channel_meta expanded map: {default=daemon} 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: expand_channel_meta expand map: {default=debug} 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: expand_channel_meta from 'debug' to 'debug' 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: expand_channel_meta expanded map: {default=debug} 2024-04-15T21:43:24.421 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 bash[24807]: f3463cf07a9237a4fbc6c53496166800dcfcb18e04896afe06467374e4e64f3f 2024-04-15T21:43:24.422 INFO:teuthology.orchestra.run.smithi007.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-15T21:43:24.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 systemd[1]: Started Ceph mon.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:43:24.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-15T21:43:24.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: monmap e1: 1 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0]} 2024-04-15T21:43:24.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: fsmap 2024-04-15T21:43:24.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: osdmap e1: 0 total, 0 up, 0 in 2024-04-15T21:43:24.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:24 smithi007 ceph-mon[24826]: mgrmap e1: no daemons active 2024-04-15T21:43:25.395 INFO:teuthology.orchestra.run.smithi007.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-15T21:43:25.395 INFO:teuthology.orchestra.run.smithi007.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-15T21:43:25.395 INFO:teuthology.orchestra.run.smithi007.stderr:Creating mgr... 2024-04-15T21:43:25.395 INFO:teuthology.orchestra.run.smithi007.stderr:Verifying port 9283 ... 2024-04-15T21:43:25.577 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Failed to reset failed state of unit ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mgr.y.service: Unit ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mgr.y.service not loaded. 2024-04-15T21:43:25.584 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: Created symlink /etc/systemd/system/ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968.target.wants/ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@.service. 2024-04-15T21:43:25.920 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:25 smithi007 systemd[1]: Starting Ceph mgr.y for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:43:26.204 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:25 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/4178339405' entity='client.admin' 2024-04-15T21:43:26.204 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:25 smithi007 podman[25041]: 2024-04-15 21:43:25.949832387 +0000 UTC m=+0.098400637 container create 1f522423be818b9bcb61c71d114e8e36f05f5025062ecdc418e81c62872c0f66 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, release=754, architecture=x86_64, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, vendor=Red Hat, Inc., GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, name=centos-stream) 2024-04-15T21:43:26.204 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:25 smithi007 podman[25041]: 2024-04-15 21:43:25.869605804 +0000 UTC m=+0.018174077 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-15T21:43:26.204 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:26 smithi007 podman[25041]: 2024-04-15 21:43:26.144087694 +0000 UTC m=+0.292655945 container init 1f522423be818b9bcb61c71d114e8e36f05f5025062ecdc418e81c62872c0f66 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, name=centos-stream, GIT_CLEAN=True, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., version=8, ceph=True, release=754, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, vcs-type=git, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-15T21:43:26.204 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:26 smithi007 podman[25041]: 2024-04-15 21:43:26.146727207 +0000 UTC m=+0.295295458 container start 1f522423be818b9bcb61c71d114e8e36f05f5025062ecdc418e81c62872c0f66 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, name=centos-stream, GIT_CLEAN=True, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-15T21:43:26.204 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:26 smithi007 bash[25041]: 1f522423be818b9bcb61c71d114e8e36f05f5025062ecdc418e81c62872c0f66 2024-04-15T21:43:26.216 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: disabled 2024-04-15T21:43:26.224 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2024-04-15T21:43:26.225 INFO:teuthology.orchestra.run.smithi007.stderr:firewalld.service is not enabled 2024-04-15T21:43:26.225 INFO:teuthology.orchestra.run.smithi007.stderr:Not possible to enable service . firewalld.service is not available 2024-04-15T21:43:26.233 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: disabled 2024-04-15T21:43:26.241 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2024-04-15T21:43:26.242 INFO:teuthology.orchestra.run.smithi007.stderr:firewalld.service is not enabled 2024-04-15T21:43:26.242 INFO:teuthology.orchestra.run.smithi007.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-15T21:43:26.242 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mgr to start... 2024-04-15T21:43:26.242 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mgr... 2024-04-15T21:43:26.464 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:26 smithi007 systemd[1]: Started Ceph mgr.y for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:43:26.738 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:26 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:26.650+0000 7fd5d563b000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T21:43:26.741 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-15T21:43:26.741 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-15T21:43:26.741 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsid": "18ed60a0-fb71-11ee-bc8f-c7b262605968", 2024-04-15T21:43:26.741 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "health": { 2024-04-15T21:43:26.741 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "checks": {}, 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mutes": [] 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum": [ 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 0 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "a" 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "monmap": { 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T21:43:26.742 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-15T21:43:26.743 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-15T21:43:26.743 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:26.743 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osdmap": { 2024-04-15T21:43:26.743 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T21:43:26.743 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-15T21:43:26.743 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-15T21:43:26.743 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-15T21:43:26.743 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-15T21:43:26.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-15T21:43:26.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-15T21:43:26.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:26.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgmap": { 2024-04-15T21:43:26.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-15T21:43:26.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-15T21:43:26.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-15T21:43:26.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-15T21:43:26.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-15T21:43:26.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-15T21:43:26.747 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsmap": { 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": false, 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modules": [ 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "iostat", 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "nfs", 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "restful" 2024-04-15T21:43:26.748 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-15T21:43:26.749 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-15T21:43:26.749 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:26.749 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "servicemap": { 2024-04-15T21:43:26.749 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T21:43:26.749 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modified": "2024-04-15T21:43:20.122777+0000", 2024-04-15T21:43:26.749 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-15T21:43:26.749 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:26.749 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-15T21:43:26.749 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-15T21:43:27.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:26 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/293196462' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T21:43:27.043 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:26 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:26.748+0000 7fd5d563b000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T21:43:27.296 INFO:teuthology.orchestra.run.smithi007.stderr:mgr not available, waiting (1/15)... 2024-04-15T21:43:27.462 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:27 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:27.370+0000 7fd5d563b000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T21:43:27.777 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:27 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:27.460+0000 7fd5d563b000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T21:43:27.777 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:27 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:27.544+0000 7fd5d563b000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T21:43:27.777 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:27 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:27.694+0000 7fd5d563b000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T21:43:27.777 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:27 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:27.775+0000 7fd5d563b000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T21:43:28.042 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:27 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:27.885+0000 7fd5d563b000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T21:43:28.043 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:27 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:27.969+0000 7fd5d563b000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T21:43:28.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:28 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:28.133+0000 7fd5d563b000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T21:43:28.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:28 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:28.219+0000 7fd5d563b000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T21:43:29.533 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:29 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:29.271+0000 7fd5d563b000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T21:43:29.533 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:29 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:29.405+0000 7fd5d563b000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T21:43:29.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:29 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:29.600+0000 7fd5d563b000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T21:43:29.793 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:29 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:29.689+0000 7fd5d563b000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T21:43:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-15T21:43:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-15T21:43:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsid": "18ed60a0-fb71-11ee-bc8f-c7b262605968", 2024-04-15T21:43:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "health": { 2024-04-15T21:43:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-15T21:43:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "checks": {}, 2024-04-15T21:43:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mutes": [] 2024-04-15T21:43:29.815 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum": [ 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 0 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "a" 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "monmap": { 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:29.816 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osdmap": { 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgmap": { 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-15T21:43:29.817 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsmap": { 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": false, 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modules": [ 2024-04-15T21:43:29.818 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "iostat", 2024-04-15T21:43:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "nfs", 2024-04-15T21:43:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "restful" 2024-04-15T21:43:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-15T21:43:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-15T21:43:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "servicemap": { 2024-04-15T21:43:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T21:43:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modified": "2024-04-15T21:43:20.122777+0000", 2024-04-15T21:43:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-15T21:43:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-15T21:43:29.819 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-15T21:43:30.107 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:29 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:29.832+0000 7fd5d563b000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T21:43:30.388 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:30 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1240271682' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T21:43:30.388 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:30 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:30.163+0000 7fd5d563b000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T21:43:30.652 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:30 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:30.386+0000 7fd5d563b000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T21:43:30.652 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:30 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:30.473+0000 7fd5d563b000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T21:43:30.652 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:30 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:30.650+0000 7fd5d563b000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T21:43:31.381 INFO:teuthology.orchestra.run.smithi007.stderr:mgr not available, waiting (2/15)... 2024-04-15T21:43:31.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:31 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:31.585+0000 7fd5d563b000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T21:43:32.188 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:31 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:31.886+0000 7fd5d563b000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T21:43:32.188 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:31 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:31.981+0000 7fd5d563b000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T21:43:32.189 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:32.134+0000 7fd5d563b000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T21:43:32.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-mon[24826]: Activating manager daemon y 2024-04-15T21:43:32.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-mon[24826]: mgrmap e2: y(active, starting, since 0.00268167s) 2024-04-15T21:43:32.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-mon[24826]: from='mgr.14100 172.21.15.7:0/200832090' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T21:43:32.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-mon[24826]: from='mgr.14100 172.21.15.7:0/200832090' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T21:43:32.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-mon[24826]: from='mgr.14100 172.21.15.7:0/200832090' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T21:43:32.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-mon[24826]: from='mgr.14100 172.21.15.7:0/200832090' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:43:32.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-mon[24826]: from='mgr.14100 172.21.15.7:0/200832090' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T21:43:32.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-mon[24826]: Manager daemon y is now available 2024-04-15T21:43:32.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-mon[24826]: from='mgr.14100 172.21.15.7:0/200832090' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:43:32.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-mon[24826]: from='mgr.14100 172.21.15.7:0/200832090' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:43:32.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-mon[24826]: from='mgr.14100 172.21.15.7:0/200832090' entity='mgr.y' 2024-04-15T21:43:32.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-mon[24826]: from='mgr.14100 172.21.15.7:0/200832090' entity='mgr.y' 2024-04-15T21:43:32.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:32 smithi007 ceph-mon[24826]: from='mgr.14100 172.21.15.7:0/200832090' entity='mgr.y' 2024-04-15T21:43:34.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:34 smithi007 ceph-mon[24826]: mgrmap e3: y(active, since 1.00642s) 2024-04-15T21:43:35.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:35 smithi007 ceph-mon[24826]: mgrmap e4: y(active, since 2s) 2024-04-15T21:43:37.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:37 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/262230283' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-15T21:43:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-15T21:43:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-15T21:43:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsid": "18ed60a0-fb71-11ee-bc8f-c7b262605968", 2024-04-15T21:43:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "health": { 2024-04-15T21:43:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-15T21:43:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "checks": {}, 2024-04-15T21:43:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mutes": [] 2024-04-15T21:43:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:37.895 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-15T21:43:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum": [ 2024-04-15T21:43:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 0 2024-04-15T21:43:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-15T21:43:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-15T21:43:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "a" 2024-04-15T21:43:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-15T21:43:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "quorum_age": 13, 2024-04-15T21:43:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "monmap": { 2024-04-15T21:43:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T21:43:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-15T21:43:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-15T21:43:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:37.896 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osdmap": { 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgmap": { 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-15T21:43:37.897 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "fsmap": { 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": true, 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modules": [ 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "iostat", 2024-04-15T21:43:37.898 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "nfs", 2024-04-15T21:43:37.899 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "restful" 2024-04-15T21:43:37.899 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ], 2024-04-15T21:43:37.899 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-15T21:43:37.899 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:37.899 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "servicemap": { 2024-04-15T21:43:37.899 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-15T21:43:37.899 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "modified": "2024-04-15T21:43:20.122777+0000", 2024-04-15T21:43:37.899 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "services": {} 2024-04-15T21:43:37.899 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: }, 2024-04-15T21:43:37.899 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-15T21:43:37.899 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-15T21:43:42.257 INFO:teuthology.orchestra.run.smithi007.stderr:mgr is available 2024-04-15T21:43:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-15T21:43:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [global] 2024-04-15T21:43:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: fsid = 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:43:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-15T21:43:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-15T21:43:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-15T21:43:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-15T21:43:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-15T21:43:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [mgr] 2024-04-15T21:43:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-15T21:43:42.864 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 2024-04-15T21:43:42.865 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: [osd] 2024-04-15T21:43:42.865 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-15T21:43:42.865 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-15T21:43:43.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:42 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/117288012' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-15T21:43:43.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:42 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/117288012' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-15T21:43:43.487 INFO:teuthology.orchestra.run.smithi007.stderr:Enabling cephadm module... 2024-04-15T21:43:44.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:44 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1139695088' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-15T21:43:44.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:44 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ignoring --setuser ceph since I am not root 2024-04-15T21:43:44.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:44 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ignoring --setgroup ceph since I am not root 2024-04-15T21:43:44.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:44 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:44.744+0000 7f882be54000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T21:43:45.110 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:44 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:44.840+0000 7f882be54000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T21:43:45.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:45 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1139695088' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-15T21:43:45.372 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:45 smithi007 ceph-mon[24826]: mgrmap e5: y(active, since 12s) 2024-04-15T21:43:45.373 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-15T21:43:45.373 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-15T21:43:45.373 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": true, 2024-04-15T21:43:45.373 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-15T21:43:45.373 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-15T21:43:45.373 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-15T21:43:45.635 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:45 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:45.455+0000 7f882be54000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T21:43:45.635 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:45 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:45.545+0000 7f882be54000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T21:43:45.635 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:45 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:45.632+0000 7f882be54000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T21:43:45.893 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:45 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:45.783+0000 7f882be54000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T21:43:45.893 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:45 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:45.865+0000 7f882be54000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T21:43:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for the mgr to restart... 2024-04-15T21:43:45.893 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mgr epoch 5... 2024-04-15T21:43:46.183 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:45 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:45.990+0000 7f882be54000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T21:43:46.184 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:46 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:46.077+0000 7f882be54000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T21:43:46.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:46 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/2104388948' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-15T21:43:46.543 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:46 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:46.245+0000 7f882be54000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T21:43:46.543 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:46 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:46.334+0000 7f882be54000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T21:43:47.706 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:47 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:47.378+0000 7f882be54000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T21:43:47.706 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:47 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:47.509+0000 7f882be54000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T21:43:47.706 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:47 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:47.703+0000 7f882be54000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T21:43:48.042 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:47 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:47.791+0000 7f882be54000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T21:43:48.043 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:47 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:47.930+0000 7f882be54000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T21:43:48.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:48 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:48.253+0000 7f882be54000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T21:43:48.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:48 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:48.478+0000 7f882be54000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T21:43:49.042 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:48 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:48.565+0000 7f882be54000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T21:43:49.043 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:48 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:48.739+0000 7f882be54000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T21:43:49.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:49 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:49.690+0000 7f882be54000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T21:43:50.253 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:49 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:49.994+0000 7f882be54000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T21:43:50.253 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:50.092+0000 7f882be54000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T21:43:50.253 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:43:50.250+0000 7f882be54000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T21:43:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-15T21:43:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-15T21:43:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "initialized": true 2024-04-15T21:43:51.361 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-15T21:43:51.543 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: [15/Apr/2024:21:43:51] ENGINE Bus STARTING 2024-04-15T21:43:51.543 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: [15/Apr/2024:21:43:51] ENGINE Serving on https://172.21.15.7:7150 2024-04-15T21:43:51.543 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: [15/Apr/2024:21:43:51] ENGINE Bus STARTED 2024-04-15T21:43:51.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: Active manager daemon y restarted 2024-04-15T21:43:51.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: Activating manager daemon y 2024-04-15T21:43:51.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: osdmap e2: 0 total, 0 up, 0 in 2024-04-15T21:43:51.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: mgrmap e6: y(active, starting, since 0.0967132s) 2024-04-15T21:43:51.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:43:51.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T21:43:51.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T21:43:51.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T21:43:51.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T21:43:51.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: Manager daemon y is now available 2024-04-15T21:43:51.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' 2024-04-15T21:43:51.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' 2024-04-15T21:43:51.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:43:51.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:43:51.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:43:51.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:43:51.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:51 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:43:51.887 INFO:teuthology.orchestra.run.smithi007.stderr:mgr epoch 5 is available 2024-04-15T21:43:51.887 INFO:teuthology.orchestra.run.smithi007.stderr:Setting orchestrator backend to cephadm... 2024-04-15T21:43:52.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:52 smithi007 ceph-mon[24826]: [15/Apr/2024:21:43:51] ENGINE Bus STARTING 2024-04-15T21:43:52.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:52 smithi007 ceph-mon[24826]: [15/Apr/2024:21:43:51] ENGINE Serving on https://172.21.15.7:7150 2024-04-15T21:43:52.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:52 smithi007 ceph-mon[24826]: [15/Apr/2024:21:43:51] ENGINE Bus STARTED 2024-04-15T21:43:52.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:52 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' 2024-04-15T21:43:52.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:52 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:43:52.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:52 smithi007 ceph-mon[24826]: mgrmap e7: y(active, since 1.09939s) 2024-04-15T21:43:53.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:53 smithi007 ceph-mon[24826]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-15T21:43:53.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:53 smithi007 ceph-mon[24826]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-15T21:43:53.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:53 smithi007 ceph-mon[24826]: mgrmap e8: y(active, since 2s) 2024-04-15T21:43:53.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:53 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' 2024-04-15T21:43:53.376 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:53 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:43:53.590 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: value unchanged 2024-04-15T21:43:54.240 INFO:teuthology.orchestra.run.smithi007.stderr:Generating ssh key... 2024-04-15T21:43:54.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:54 smithi007 ceph-mon[24826]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:43:55.377 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-mon[24826]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:43:55.377 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' 2024-04-15T21:43:55.377 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: Generating public/private rsa key pair. 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: Your identification has been saved in /tmp/tmpeo16wm5j/key. 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: Your public key has been saved in /tmp/tmpeo16wm5j/key.pub. 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: The key fingerprint is: 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: SHA256:eWq2tjtb5uI06jOC19IYPhn0nphvLFpU5cfvzE9M0RI ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: The key's randomart image is: 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: +---[RSA 3072]----+ 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: | . E | 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: | o . o | 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: | . . o o .| 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: | .. o . o | 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: | ... S . . . | 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: | .o . o + o | 2024-04-15T21:43:55.378 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: | o.% .* o + o | 2024-04-15T21:43:55.379 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: | ..% X=+* o | 2024-04-15T21:43:55.379 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: | .o O+=**. . | 2024-04-15T21:43:55.379 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:43:55 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: +----[SHA256]-----+ 2024-04-15T21:43:56.234 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgTihs3Ma3yUGf6mqL/pvpnbtL0ws7nj0vkjzPQm4xvt/2pLsr4W0J61prkfhgDlCnv+U8kGmYNHfq1f4Z3uLM0LnbXIMj5MsD0tB991jU00Tjx2cBvRSLAuyZrcgsM5yMe2ht9CoLbl96OZb8zYYWP73STWVs3EMz5bpoeiWmAlQlB4thLSpCJi6bRoOiXyBt8SAfjiLRkq9kYCYsk0s8OVAUldnMhWddQ85sFGqka5i/Fnj7sJZG2GlvBlVrNmn8PQCOSZoZo6Nps0BCIbJw9tYK/CA+zfCj/hE+Iq2jakASz1pEOnAd88G3YKXMaeJbvGH5qd8RhVK5I0camJep83g82pwfIWdwICx2F/OpEzn5hPbjE6jtJh+bv4lmFbNwM+VYttZimam7UCoMW6U5R+RTsZB1FI838LKQUknFku6ddwL299YnL9GQ58vd4E35xkHl7QHWrLf80TXy46wVomAAGSl7SAHAF0IJdXX/jmWexeMdqJSFpQDuOdLrPvE= ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:43:56.525 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:56 smithi007 ceph-mon[24826]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:43:56.526 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:56 smithi007 ceph-mon[24826]: Generating ssh key... 2024-04-15T21:43:56.779 INFO:teuthology.orchestra.run.smithi007.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-15T21:43:56.779 INFO:teuthology.orchestra.run.smithi007.stderr:Adding key to root@localhost authorized_keys... 2024-04-15T21:43:56.779 INFO:teuthology.orchestra.run.smithi007.stderr:Adding host smithi007... 2024-04-15T21:43:57.458 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:57 smithi007 ceph-mon[24826]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:43:58.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:58 smithi007 ceph-mon[24826]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi007", "addr": "172.21.15.7", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:43:58.482 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:58 smithi007 ceph-mon[24826]: Deploying cephadm binary to smithi007 2024-04-15T21:43:58.482 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: Added host 'smithi007' with addr '172.21.15.7' 2024-04-15T21:43:59.010 INFO:teuthology.orchestra.run.smithi007.stderr:Deploying unmanaged mon service... 2024-04-15T21:43:59.606 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-15T21:43:59.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:59 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' 2024-04-15T21:43:59.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:59 smithi007 ceph-mon[24826]: Added host smithi007 2024-04-15T21:43:59.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:43:59 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:44:00.244 INFO:teuthology.orchestra.run.smithi007.stderr:Deploying unmanaged mgr service... 2024-04-15T21:44:00.946 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:00 smithi007 ceph-mon[24826]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:44:00.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:00 smithi007 ceph-mon[24826]: Saving service mon spec with placement count:5 2024-04-15T21:44:00.947 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:00 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' 2024-04-15T21:44:01.072 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-15T21:44:02.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:02 smithi007 ceph-mon[24826]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:44:02.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:02 smithi007 ceph-mon[24826]: Saving service mgr spec with placement count:2 2024-04-15T21:44:02.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:02 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' 2024-04-15T21:44:02.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:02 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' 2024-04-15T21:44:03.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:03 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' 2024-04-15T21:44:03.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:03 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/117078886' entity='client.admin' 2024-04-15T21:44:04.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:04 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' 2024-04-15T21:44:04.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:04 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/553712073' entity='client.admin' 2024-04-15T21:44:04.984 INFO:teuthology.orchestra.run.smithi007.stderr:Enabling the dashboard module... 2024-04-15T21:44:06.741 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:06 smithi007 ceph-mon[24826]: from='mgr.14118 172.21.15.7:0/1683374358' entity='mgr.y' 2024-04-15T21:44:06.742 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:06 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/2235804134' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-15T21:44:06.742 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:06 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ignoring --setuser ceph since I am not root 2024-04-15T21:44:06.742 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:06 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ignoring --setgroup ceph since I am not root 2024-04-15T21:44:07.293 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:07 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:07.011+0000 7f3044b45000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T21:44:07.293 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:07 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:07.102+0000 7f3044b45000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T21:44:07.596 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:07 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/2235804134' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-15T21:44:07.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:07 smithi007 ceph-mon[24826]: mgrmap e9: y(active, since 16s) 2024-04-15T21:44:07.855 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:07 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:07.729+0000 7f3044b45000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T21:44:07.855 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:07 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:07.820+0000 7f3044b45000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T21:44:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-15T21:44:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-15T21:44:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "available": true, 2024-04-15T21:44:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "active_name": "y", 2024-04-15T21:44:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-15T21:44:07.856 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-15T21:44:08.150 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:07 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:07.913+0000 7f3044b45000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T21:44:08.151 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:08 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:08.064+0000 7f3044b45000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T21:44:08.151 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:08 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:08.147+0000 7f3044b45000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T21:44:08.378 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for the mgr to restart... 2024-04-15T21:44:08.378 INFO:teuthology.orchestra.run.smithi007.stderr:Waiting for mgr epoch 9... 2024-04-15T21:44:08.496 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:08 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:08.260+0000 7f3044b45000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T21:44:08.496 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:08 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:08.346+0000 7f3044b45000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T21:44:08.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:08 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/2127850740' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-15T21:44:08.793 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:08 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:08.516+0000 7f3044b45000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T21:44:08.793 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:08 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:08.604+0000 7f3044b45000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T21:44:09.996 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:09 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:09.665+0000 7f3044b45000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T21:44:09.996 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:09 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:09.799+0000 7f3044b45000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T21:44:09.996 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:09 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:09.993+0000 7f3044b45000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T21:44:10.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:10 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:10.082+0000 7f3044b45000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T21:44:10.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:10 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:10.221+0000 7f3044b45000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T21:44:10.859 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:10 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:10.544+0000 7f3044b45000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T21:44:10.859 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:10 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:10.769+0000 7f3044b45000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T21:44:10.859 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:10 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:10.855+0000 7f3044b45000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T21:44:11.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:11 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:11.030+0000 7f3044b45000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T21:44:12.275 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:11 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:11.969+0000 7f3044b45000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T21:44:12.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:12 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:12.273+0000 7f3044b45000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T21:44:12.543 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:12 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:12.368+0000 7f3044b45000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T21:44:12.543 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:12 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:44:12.522+0000 7f3044b45000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T21:44:13.588 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: { 2024-04-15T21:44:13.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: Active manager daemon y restarted 2024-04-15T21:44:13.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: Activating manager daemon y 2024-04-15T21:44:13.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: osdmap e3: 0 total, 0 up, 0 in 2024-04-15T21:44:13.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: mgrmap e10: y(active, starting, since 0.0525799s) 2024-04-15T21:44:13.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:44:13.589 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T21:44:13.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T21:44:13.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T21:44:13.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T21:44:13.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: Manager daemon y is now available 2024-04-15T21:44:13.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:44:13.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:44:13.590 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:44:13.591 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:44:13.592 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-15T21:44:13.592 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: "initialized": true 2024-04-15T21:44:13.592 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: } 2024-04-15T21:44:13.985 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: [15/Apr/2024:21:44:13] ENGINE Bus STARTING 2024-04-15T21:44:13.985 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: [15/Apr/2024:21:44:13] ENGINE Serving on https://172.21.15.7:7150 2024-04-15T21:44:13.985 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:44:13 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: [15/Apr/2024:21:44:13] ENGINE Bus STARTED 2024-04-15T21:44:14.247 INFO:teuthology.orchestra.run.smithi007.stderr:mgr epoch 9 is available 2024-04-15T21:44:14.247 INFO:teuthology.orchestra.run.smithi007.stderr:Generating a dashboard self-signed certificate... 2024-04-15T21:44:14.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:14 smithi007 ceph-mon[24826]: mgrmap e11: y(active, since 1.05501s) 2024-04-15T21:44:14.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:14 smithi007 ceph-mon[24826]: [15/Apr/2024:21:44:13] ENGINE Bus STARTING 2024-04-15T21:44:14.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:14 smithi007 ceph-mon[24826]: [15/Apr/2024:21:44:13] ENGINE Serving on https://172.21.15.7:7150 2024-04-15T21:44:14.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:14 smithi007 ceph-mon[24826]: [15/Apr/2024:21:44:13] ENGINE Bus STARTED 2024-04-15T21:44:14.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:14 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:14.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:14 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:15.130 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-15T21:44:15.712 INFO:teuthology.orchestra.run.smithi007.stderr:Creating initial admin user... 2024-04-15T21:44:16.378 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:16 smithi007 ceph-mon[24826]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:44:16.379 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:16 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:16.379 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:16 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:16.379 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:16 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:16.379 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:16 smithi007 ceph-mon[24826]: mgrmap e12: y(active, since 2s) 2024-04-15T21:44:16.630 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$rty7zo1Lv2DZxybizsaX7OnkE.P3/1/uIQb6ns00ijK1kTN0DSTum", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713217456, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-15T21:44:17.371 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:17 smithi007 ceph-mon[24826]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:44:17.371 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:17 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:17.374 INFO:teuthology.orchestra.run.smithi007.stderr:Fetching dashboard port number... 2024-04-15T21:44:18.453 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: 8443 2024-04-15T21:44:18.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:18 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3344409515' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-15T21:44:18.993 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: disabled 2024-04-15T21:44:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:systemctl: inactive 2024-04-15T21:44:19.001 INFO:teuthology.orchestra.run.smithi007.stderr:firewalld.service is not enabled 2024-04-15T21:44:19.002 INFO:teuthology.orchestra.run.smithi007.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-15T21:44:19.004 INFO:teuthology.orchestra.run.smithi007.stderr:Ceph Dashboard is now available at: 2024-04-15T21:44:19.004 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-15T21:44:19.004 INFO:teuthology.orchestra.run.smithi007.stderr: URL: https://smithi007.front.sepia.ceph.com:8443/ 2024-04-15T21:44:19.004 INFO:teuthology.orchestra.run.smithi007.stderr: User: admin 2024-04-15T21:44:19.004 INFO:teuthology.orchestra.run.smithi007.stderr: Password: tnjxyl4wi2 2024-04-15T21:44:19.004 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-15T21:44:19.004 INFO:teuthology.orchestra.run.smithi007.stderr:Enabling autotune for osd_memory_target 2024-04-15T21:44:20.742 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:20 smithi007 ceph-mon[24826]: mgrmap e13: y(active, since 6s) 2024-04-15T21:44:21.059 INFO:teuthology.orchestra.run.smithi007.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-15T21:44:21.623 INFO:teuthology.orchestra.run.smithi007.stderr:You can access the Ceph CLI with: 2024-04-15T21:44:21.623 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-15T21:44:21.623 INFO:teuthology.orchestra.run.smithi007.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-15T21:44:21.623 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-15T21:44:21.623 INFO:teuthology.orchestra.run.smithi007.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-15T21:44:21.623 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-15T21:44:21.623 INFO:teuthology.orchestra.run.smithi007.stderr: ceph telemetry on 2024-04-15T21:44:21.623 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-15T21:44:21.623 INFO:teuthology.orchestra.run.smithi007.stderr:For more information see: 2024-04-15T21:44:21.623 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-15T21:44:21.623 INFO:teuthology.orchestra.run.smithi007.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-15T21:44:21.624 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-15T21:44:21.624 INFO:teuthology.orchestra.run.smithi007.stderr:Bootstrap complete. 2024-04-15T21:44:21.661 INFO:tasks.cephadm:Fetching config... 2024-04-15T21:44:21.661 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-15T21:44:21.661 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-15T21:44:21.678 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-15T21:44:21.678 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-15T21:44:21.678 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-15T21:44:21.733 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-15T21:44:21.734 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-15T21:44:21.734 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-15T21:44:21.802 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-15T21:44:21.803 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-15T21:44:21.803 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-15T21:44:21.858 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-15T21:44:21.858 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgTihs3Ma3yUGf6mqL/pvpnbtL0ws7nj0vkjzPQm4xvt/2pLsr4W0J61prkfhgDlCnv+U8kGmYNHfq1f4Z3uLM0LnbXIMj5MsD0tB991jU00Tjx2cBvRSLAuyZrcgsM5yMe2ht9CoLbl96OZb8zYYWP73STWVs3EMz5bpoeiWmAlQlB4thLSpCJi6bRoOiXyBt8SAfjiLRkq9kYCYsk0s8OVAUldnMhWddQ85sFGqka5i/Fnj7sJZG2GlvBlVrNmn8PQCOSZoZo6Nps0BCIbJw9tYK/CA+zfCj/hE+Iq2jakASz1pEOnAd88G3YKXMaeJbvGH5qd8RhVK5I0camJep83g82pwfIWdwICx2F/OpEzn5hPbjE6jtJh+bv4lmFbNwM+VYttZimam7UCoMW6U5R+RTsZB1FI838LKQUknFku6ddwL299YnL9GQ58vd4E35xkHl7QHWrLf80TXy46wVomAAGSl7SAHAF0IJdXX/jmWexeMdqJSFpQDuOdLrPvE= ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-15T21:44:21.942 INFO:teuthology.orchestra.run.smithi007.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgTihs3Ma3yUGf6mqL/pvpnbtL0ws7nj0vkjzPQm4xvt/2pLsr4W0J61prkfhgDlCnv+U8kGmYNHfq1f4Z3uLM0LnbXIMj5MsD0tB991jU00Tjx2cBvRSLAuyZrcgsM5yMe2ht9CoLbl96OZb8zYYWP73STWVs3EMz5bpoeiWmAlQlB4thLSpCJi6bRoOiXyBt8SAfjiLRkq9kYCYsk0s8OVAUldnMhWddQ85sFGqka5i/Fnj7sJZG2GlvBlVrNmn8PQCOSZoZo6Nps0BCIbJw9tYK/CA+zfCj/hE+Iq2jakASz1pEOnAd88G3YKXMaeJbvGH5qd8RhVK5I0camJep83g82pwfIWdwICx2F/OpEzn5hPbjE6jtJh+bv4lmFbNwM+VYttZimam7UCoMW6U5R+RTsZB1FI838LKQUknFku6ddwL299YnL9GQ58vd4E35xkHl7QHWrLf80TXy46wVomAAGSl7SAHAF0IJdXX/jmWexeMdqJSFpQDuOdLrPvE= ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:44:21.957 DEBUG:teuthology.orchestra.run.smithi033:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgTihs3Ma3yUGf6mqL/pvpnbtL0ws7nj0vkjzPQm4xvt/2pLsr4W0J61prkfhgDlCnv+U8kGmYNHfq1f4Z3uLM0LnbXIMj5MsD0tB991jU00Tjx2cBvRSLAuyZrcgsM5yMe2ht9CoLbl96OZb8zYYWP73STWVs3EMz5bpoeiWmAlQlB4thLSpCJi6bRoOiXyBt8SAfjiLRkq9kYCYsk0s8OVAUldnMhWddQ85sFGqka5i/Fnj7sJZG2GlvBlVrNmn8PQCOSZoZo6Nps0BCIbJw9tYK/CA+zfCj/hE+Iq2jakASz1pEOnAd88G3YKXMaeJbvGH5qd8RhVK5I0camJep83g82pwfIWdwICx2F/OpEzn5hPbjE6jtJh+bv4lmFbNwM+VYttZimam7UCoMW6U5R+RTsZB1FI838LKQUknFku6ddwL299YnL9GQ58vd4E35xkHl7QHWrLf80TXy46wVomAAGSl7SAHAF0IJdXX/jmWexeMdqJSFpQDuOdLrPvE= ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-15T21:44:22.006 INFO:teuthology.orchestra.run.smithi033.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDgTihs3Ma3yUGf6mqL/pvpnbtL0ws7nj0vkjzPQm4xvt/2pLsr4W0J61prkfhgDlCnv+U8kGmYNHfq1f4Z3uLM0LnbXIMj5MsD0tB991jU00Tjx2cBvRSLAuyZrcgsM5yMe2ht9CoLbl96OZb8zYYWP73STWVs3EMz5bpoeiWmAlQlB4thLSpCJi6bRoOiXyBt8SAfjiLRkq9kYCYsk0s8OVAUldnMhWddQ85sFGqka5i/Fnj7sJZG2GlvBlVrNmn8PQCOSZoZo6Nps0BCIbJw9tYK/CA+zfCj/hE+Iq2jakASz1pEOnAd88G3YKXMaeJbvGH5qd8RhVK5I0camJep83g82pwfIWdwICx2F/OpEzn5hPbjE6jtJh+bv4lmFbNwM+VYttZimam7UCoMW6U5R+RTsZB1FI838LKQUknFku6ddwL299YnL9GQ58vd4E35xkHl7QHWrLf80TXy46wVomAAGSl7SAHAF0IJdXX/jmWexeMdqJSFpQDuOdLrPvE= ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:44:22.022 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-15T21:44:22.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:22 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1945237990' entity='client.admin' 2024-04-15T21:44:25.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:24 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:25.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:24 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi007", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:44:25.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:24 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:25.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:24 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:44:25.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:24 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:25.909 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-15T21:44:25.909 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-15T21:44:26.208 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:25 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/699751379' entity='client.admin' 2024-04-15T21:44:26.208 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:25 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:44:26.208 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:25 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:28.420 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi033 2024-04-15T21:44:28.420 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-15T21:44:28.420 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/etc/ceph/ceph.conf 2024-04-15T21:44:28.439 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-15T21:44:28.439 DEBUG:teuthology.orchestra.run.smithi033:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T21:44:28.498 INFO:tasks.cephadm:Adding host smithi033 to orchestrator... 2024-04-15T21:44:28.498 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch host add smithi033 2024-04-15T21:44:28.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:28 smithi007 ceph-mon[24826]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:44:28.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:28 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:28.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:28 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:44:28.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:28 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:44:28.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:28 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:44:28.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:28 smithi007 ceph-mon[24826]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-15T21:44:28.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:28 smithi007 ceph-mon[24826]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-15T21:44:28.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:28 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:28.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:28 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:31.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:30 smithi007 ceph-mon[24826]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi033", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:44:31.352 INFO:teuthology.orchestra.run.smithi007.stdout:Added host 'smithi033' with addr '172.21.15.33' 2024-04-15T21:44:31.923 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch host ls --format=json 2024-04-15T21:44:32.523 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:32 smithi007 ceph-mon[24826]: Deploying cephadm binary to smithi033 2024-04-15T21:44:32.523 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:32 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:32.523 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:32 smithi007 ceph-mon[24826]: Added host smithi033 2024-04-15T21:44:32.523 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:32 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:44:32.523 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:32 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:44:32.523 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:32 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:44:32.523 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:32 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:32.523 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:32 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:33.768 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:44:33.768 INFO:teuthology.orchestra.run.smithi007.stdout:[{"addr": "172.21.15.7", "hostname": "smithi007", "labels": [], "status": ""}, {"addr": "172.21.15.33", "hostname": "smithi033", "labels": [], "status": ""}] 2024-04-15T21:44:34.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:33 smithi007 ceph-mon[24826]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:34.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:33 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:34.383 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-15T21:44:34.383 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd crush tunables default 2024-04-15T21:44:35.630 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:35 smithi007 ceph-mon[24826]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T21:44:36.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:36 smithi007 ceph-mon[24826]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:36.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:36 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:36.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:36 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:36.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:36 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi033", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:44:36.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:36 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:36.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:36 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1380829222' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-15T21:44:36.885 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-04-15T21:44:37.439 INFO:tasks.cephadm:Adding mon.a on smithi007 2024-04-15T21:44:37.439 INFO:tasks.cephadm:Adding mon.c on smithi007 2024-04-15T21:44:37.439 INFO:tasks.cephadm:Adding mon.b on smithi033 2024-04-15T21:44:37.440 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch apply mon '3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi033:172.21.15.33=b' 2024-04-15T21:44:38.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:37 smithi007 ceph-mon[24826]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:38.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:37 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1380829222' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-15T21:44:38.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:37 smithi007 ceph-mon[24826]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T21:44:39.237 INFO:teuthology.orchestra.run.smithi033.stdout:Scheduled mon update... 2024-04-15T21:44:39.963 DEBUG:teuthology.orchestra.run.smithi007:mon.c> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.c.service 2024-04-15T21:44:39.965 DEBUG:teuthology.orchestra.run.smithi033:mon.b> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.b.service 2024-04-15T21:44:39.970 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-15T21:44:39.970 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T21:44:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:40 smithi007 ceph-mon[24826]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:40 smithi007 ceph-mon[24826]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi033:172.21.15.33=b", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:44:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:40 smithi007 ceph-mon[24826]: Saving service mon spec with placement smithi007:172.21.15.7=a;smithi007:[v2:172.21.15.7:3301,v1:172.21.15.7:6790]=c;smithi033:172.21.15.33=b;count:3 2024-04-15T21:44:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:44:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:44:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:44:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:40 smithi007 ceph-mon[24826]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-15T21:44:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:40 smithi007 ceph-mon[24826]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-04-15T21:44:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:40.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T21:44:40.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:44:41.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:41 smithi007 ceph-mon[24826]: Deploying daemon mon.c on smithi007 2024-04-15T21:44:41.888 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-15T21:44:41.888 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":1,"fsid":"18ed60a0-fb71-11ee-bc8f-c7b262605968","modified":"2024-04-15T21:43:17.375905Z","created":"2024-04-15T21:43:17.375905Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-15T21:44:41.890 INFO:teuthology.orchestra.run.smithi033.stderr:dumped monmap epoch 1 2024-04-15T21:44:42.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[24826]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:42.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[24826]: from='client.? 172.21.15.33:0/2677011753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T21:44:42.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 podman[29523]: 2024-04-15 21:44:42.479585764 +0000 UTC m=+0.116687413 container create ee73c9e2f8d75b1c6ce0f7080f2e8f8741d2da412531170b2df570c0b7629fa0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-c, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., architecture=x86_64, vcs-type=git, io.openshift.tags=base centos centos-stream, version=8, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , distribution-scope=public, name=centos-stream) 2024-04-15T21:44:43.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 podman[29523]: 2024-04-15 21:44:42.690247426 +0000 UTC m=+0.327349075 container init ee73c9e2f8d75b1c6ce0f7080f2e8f8741d2da412531170b2df570c0b7629fa0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-c, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, ceph=True, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754) 2024-04-15T21:44:43.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 podman[29523]: 2024-04-15 21:44:42.693455572 +0000 UTC m=+0.330557222 container start ee73c9e2f8d75b1c6ce0f7080f2e8f8741d2da412531170b2df570c0b7629fa0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-c, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, com.redhat.component=centos-stream-container, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, version=8, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.openshift.expose-services=, ceph=True, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, release=754) 2024-04-15T21:44:43.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: set uid:gid to 167:167 (ceph:ceph) 2024-04-15T21:44:43.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-15T21:44:43.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: pidfile_write: ignore empty --pid-file 2024-04-15T21:44:43.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: load: jerasure load: lrc 2024-04-15T21:44:43.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: RocksDB version: 6.15.5 2024-04-15T21:44:43.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-15T21:44:43.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Compile date Apr 18 2022 2024-04-15T21:44:43.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: DB SUMMARY 2024-04-15T21:44:43.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: DB Session ID: ALCYINKL7JLKLR6ZPP5J 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: CURRENT file: CURRENT 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: IDENTITY file: IDENTITY 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.error_if_exists: 0 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.create_if_missing: 0 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.paranoid_checks: 1 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.env: 0x55c5518c6860 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.fs: Posix File System 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.info_log: 0x55c553cbfdc0 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_file_opening_threads: 16 2024-04-15T21:44:43.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.statistics: (nil) 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.use_fsync: 0 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_log_file_size: 0 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.keep_log_file_num: 1000 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.recycle_log_file_num: 0 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.allow_fallocate: 1 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.allow_mmap_reads: 0 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.allow_mmap_writes: 0 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.use_direct_reads: 0 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.create_missing_column_families: 0 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.db_log_dir: 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-04-15T21:44:43.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-15T21:44:43.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-15T21:44:43.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-15T21:44:43.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-15T21:44:43.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-15T21:44:43.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-15T21:44:43.047 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.advise_random_on_open: 1 2024-04-15T21:44:43.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.db_write_buffer_size: 0 2024-04-15T21:44:43.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.write_buffer_manager: 0x55c553db02d0 2024-04-15T21:44:43.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-15T21:44:43.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-15T21:44:43.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-15T21:44:43.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-15T21:44:43.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.rate_limiter: (nil) 2024-04-15T21:44:43.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-15T21:44:43.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.wal_recovery_mode: 2 2024-04-15T21:44:43.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.enable_thread_tracking: 0 2024-04-15T21:44:43.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.enable_pipelined_write: 0 2024-04-15T21:44:43.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.unordered_write: 0 2024-04-15T21:44:43.048 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.row_cache: None 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.wal_filter: None 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.allow_ingest_behind: 0 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.preserve_deletes: 0 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.two_write_queues: 0 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.manual_wal_flush: 0 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.atomic_flush: 0 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-15T21:44:43.049 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.log_readahead_size: 0 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.best_efforts_recovery: 0 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.allow_data_in_errors: 0 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.db_host_id: __hostname__ 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_background_jobs: 2 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_background_compactions: -1 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_subcompactions: 1 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-15T21:44:43.050 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_total_wal_size: 0 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_open_files: -1 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.bytes_per_sync: 0 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compaction_readahead_size: 0 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_background_flushes: -1 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Compression algorithms supported: 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: kZSTD supported: 0 2024-04-15T21:44:43.051 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: kXpressCompression supported: 0 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: kLZ4HCCompression supported: 1 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: kLZ4Compression supported: 1 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: kBZip2Compression supported: 0 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: kZlibCompression supported: 1 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: kSnappyCompression supported: 1 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.merge_operator: 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compaction_filter: None 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compaction_filter_factory: None 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.sst_partitioner_factory: None 2024-04-15T21:44:43.052 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c553c8bd00) 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout: cache_index_and_filter_blocks: 1 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout: pin_top_level_index_and_filter: 1 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_type: 0 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout: data_block_index_type: 0 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_shortening: 1 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout: hash_index_allow_collision: 1 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout: checksum: 1 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout: no_block_cache: 0 2024-04-15T21:44:43.053 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache: 0x55c553cf6170 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_name: BinnedLRUCache 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_options: 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: capacity : 536870912 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: num_shard_bits : 4 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: strict_capacity_limit : 0 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: high_pri_pool_ratio: 0.000 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_cache_compressed: (nil) 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: persistent_cache: (nil) 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_size: 4096 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_size_deviation: 10 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_restart_interval: 16 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: index_block_restart_interval: 1 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: metadata_block_size: 4096 2024-04-15T21:44:43.054 INFO:journalctl@ceph.mon.c.smithi007.stdout: partition_filters: 0 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout: use_delta_encoding: 1 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout: whole_key_filtering: 1 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout: verify_compression: 0 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout: read_amp_bytes_per_bit: 0 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout: format_version: 4 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout: enable_index_compression: 1 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout: block_align: 0 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.write_buffer_size: 33554432 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_write_buffer_number: 2 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compression: NoCompression 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.bottommost_compression: Disabled 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.prefix_extractor: nullptr 2024-04-15T21:44:43.055 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.num_levels: 7 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compression_opts.level: 32767 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compression_opts.strategy: 0 2024-04-15T21:44:43.056 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compression_opts.enabled: false 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.target_file_size_base: 67108864 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-15T21:44:43.057 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.arena_block_size: 4194304 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.disable_auto_compactions: 0 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-15T21:44:43.058 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.table_properties_collectors: 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.inplace_update_support: 0 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.bloom_locality: 0 2024-04-15T21:44:43.059 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.max_successive_merges: 0 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.paranoid_file_checks: 0 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.force_consistency_checks: 1 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.report_bg_io_stats: 0 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.ttl: 2592000 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.enable_blob_files: false 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.min_blob_size: 0 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.blob_file_size: 268435456 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-15T21:44:43.060 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713217482719622, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713217482719919, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713217482, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "403d7818-0808-4405-bc25-d8828f63acee", "db_session_id": "ALCYINKL7JLKLR6ZPP5J"}} 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713217482720244, "job": 1, "event": "recovery_finished"} 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55c553cdca80 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: DB pointer 0x55c553cec000 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** DB Stats ** 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-15T21:44:43.061 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-15T21:44:43.062 INFO:journalctl@ceph.mon.c.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-15T21:44:43.063 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** Compaction Stats [default] ** 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T21:44:43.064 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: using public_addrv [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: starting mon.c rank -1 at public addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] at bind addrs [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(???) e0 preinit fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing).mds e1 new map 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout: e1 2024-04-15T21:44:43.065 INFO:journalctl@ceph.mon.c.smithi007.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout: legacy client fscid: -1 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout: 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout: No filesystems configured 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T21:44:43.066 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:43.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: from='client.? 172.21.15.33:0/2677011753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T21:44:43.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-15T21:44:43.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 ceph-mon[29541]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-15T21:44:43.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 bash[29523]: ee73c9e2f8d75b1c6ce0f7080f2e8f8741d2da412531170b2df570c0b7629fa0 2024-04-15T21:44:43.067 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:42 smithi007 systemd[1]: Started Ceph mon.c for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:44:43.443 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-15T21:44:43.444 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T21:44:48.243 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-15T21:44:48.243 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":2,"fsid":"18ed60a0-fb71-11ee-bc8f-c7b262605968","modified":"2024-04-15T21:44:42.738086Z","created":"2024-04-15T21:43:17.375905Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3301","nonce":0},{"type":"v1","addr":"172.21.15.7:6790","nonce":0}]},"addr":"172.21.15.7:6790/0","public_addr":"172.21.15.7:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-15T21:44:48.244 INFO:teuthology.orchestra.run.smithi033.stderr:dumped monmap epoch 2 2024-04-15T21:44:48.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:48.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:44:48.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: mon.a calling monitor election 2024-04-15T21:44:48.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:48.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:48.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:48.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:48.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: mon.c calling monitor election 2024-04-15T21:44:48.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:48.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:48.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:48.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:48.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T21:44:48.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-15T21:44:48.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: fsmap 2024-04-15T21:44:48.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T21:44:48.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: mgrmap e13: y(active, since 35s) 2024-04-15T21:44:48.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: overall HEALTH_OK 2024-04-15T21:44:48.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:48.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T21:44:48.295 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:44:48.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:48.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:44:48.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: mon.a calling monitor election 2024-04-15T21:44:48.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:48.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:48.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:48.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:48.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: mon.c calling monitor election 2024-04-15T21:44:48.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:48.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:48.295 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:48.296 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:48.296 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T21:44:48.296 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-15T21:44:48.296 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: fsmap 2024-04-15T21:44:48.296 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T21:44:48.296 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: mgrmap e13: y(active, since 35s) 2024-04-15T21:44:48.296 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: overall HEALTH_OK 2024-04-15T21:44:48.296 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:48.296 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T21:44:48.296 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:47 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:44:49.196 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:48 smithi007 ceph-mon[24826]: Deploying daemon mon.b on smithi033 2024-04-15T21:44:49.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:48 smithi007 ceph-mon[24826]: from='client.? 172.21.15.33:0/451700032' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T21:44:49.197 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:48 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:49.197 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:48 smithi007 ceph-mon[29541]: Deploying daemon mon.b on smithi033 2024-04-15T21:44:49.197 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:48 smithi007 ceph-mon[29541]: from='client.? 172.21.15.33:0/451700032' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T21:44:49.197 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:48 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:50.022 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-15T21:44:50.022 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T21:44:50.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:50 smithi007 ceph-mon[24826]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:50.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:50 smithi007 ceph-mon[29541]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:52.541 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-15T21:44:52.541 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":2,"fsid":"18ed60a0-fb71-11ee-bc8f-c7b262605968","modified":"2024-04-15T21:44:42.738086Z","created":"2024-04-15T21:43:17.375905Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3301","nonce":0},{"type":"v1","addr":"172.21.15.7:6790","nonce":0}]},"addr":"172.21.15.7:6790/0","public_addr":"172.21.15.7:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-15T21:44:52.543 INFO:teuthology.orchestra.run.smithi033.stderr:dumped monmap epoch 2 2024-04-15T21:44:52.595 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 podman[25619]: 2024-04-15 21:44:52.572096767 +0000 UTC m=+0.279648804 container start f5d803c8b945f1e57f66466add6390db6777963b7379919e4ff4f3eb0b9aa1a7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-b, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, name=centos-stream, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , RELEASE=HEAD, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, release=754, io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, vendor=Red Hat, Inc.) 2024-04-15T21:44:52.595 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: set uid:gid to 167:167 (ceph:ceph) 2024-04-15T21:44:52.595 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-15T21:44:52.595 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: pidfile_write: ignore empty --pid-file 2024-04-15T21:44:52.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:52 smithi007 ceph-mon[24826]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:52.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:52 smithi007 ceph-mon[29541]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:53.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: load: jerasure load: lrc 2024-04-15T21:44:53.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: RocksDB version: 6.15.5 2024-04-15T21:44:53.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-15T21:44:53.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Compile date Apr 18 2022 2024-04-15T21:44:53.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: DB SUMMARY 2024-04-15T21:44:53.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: DB Session ID: J99BV8R4HK00DU7D8AWF 2024-04-15T21:44:53.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: CURRENT file: CURRENT 2024-04-15T21:44:53.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: IDENTITY file: IDENTITY 2024-04-15T21:44:53.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-15T21:44:53.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-15T21:44:53.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-15T21:44:53.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.error_if_exists: 0 2024-04-15T21:44:53.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.create_if_missing: 0 2024-04-15T21:44:53.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.paranoid_checks: 1 2024-04-15T21:44:53.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-15T21:44:53.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.env: 0x558dfe0c6860 2024-04-15T21:44:53.026 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.fs: Posix File System 2024-04-15T21:44:53.026 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.info_log: 0x558dffd5fdc0 2024-04-15T21:44:53.026 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_file_opening_threads: 16 2024-04-15T21:44:53.026 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.statistics: (nil) 2024-04-15T21:44:53.026 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.use_fsync: 0 2024-04-15T21:44:53.026 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_log_file_size: 0 2024-04-15T21:44:53.026 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-15T21:44:53.026 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-15T21:44:53.026 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.keep_log_file_num: 1000 2024-04-15T21:44:53.026 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.recycle_log_file_num: 0 2024-04-15T21:44:53.026 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.allow_fallocate: 1 2024-04-15T21:44:53.026 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.allow_mmap_reads: 0 2024-04-15T21:44:53.026 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.allow_mmap_writes: 0 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.use_direct_reads: 0 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.create_missing_column_families: 0 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.db_log_dir: 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.advise_random_on_open: 1 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.db_write_buffer_size: 0 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.write_buffer_manager: 0x558dffe50240 2024-04-15T21:44:53.027 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.rate_limiter: (nil) 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.wal_recovery_mode: 2 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.enable_thread_tracking: 0 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.enable_pipelined_write: 0 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.unordered_write: 0 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-15T21:44:53.028 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.row_cache: None 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.wal_filter: None 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.allow_ingest_behind: 0 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.preserve_deletes: 0 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.two_write_queues: 0 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.manual_wal_flush: 0 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.atomic_flush: 0 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.log_readahead_size: 0 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.best_efforts_recovery: 0 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-15T21:44:53.029 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-15T21:44:53.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.allow_data_in_errors: 0 2024-04-15T21:44:53.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.db_host_id: __hostname__ 2024-04-15T21:44:53.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_background_jobs: 2 2024-04-15T21:44:53.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_background_compactions: -1 2024-04-15T21:44:53.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_subcompactions: 1 2024-04-15T21:44:53.030 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-15T21:44:53.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-15T21:44:53.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-15T21:44:53.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_total_wal_size: 0 2024-04-15T21:44:53.031 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-15T21:44:53.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-15T21:44:53.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-15T21:44:53.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-15T21:44:53.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_open_files: -1 2024-04-15T21:44:53.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.bytes_per_sync: 0 2024-04-15T21:44:53.032 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compaction_readahead_size: 0 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_background_flushes: -1 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Compression algorithms supported: 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: kZSTD supported: 0 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: kXpressCompression supported: 0 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: kLZ4HCCompression supported: 1 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: kLZ4Compression supported: 1 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: kBZip2Compression supported: 0 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: kZlibCompression supported: 1 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: kSnappyCompression supported: 1 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-15T21:44:53.033 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-04-15T21:44:53.034 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-15T21:44:53.034 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-15T21:44:53.034 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.merge_operator: 2024-04-15T21:44:53.034 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compaction_filter: None 2024-04-15T21:44:53.034 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compaction_filter_factory: None 2024-04-15T21:44:53.034 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.sst_partitioner_factory: None 2024-04-15T21:44:53.034 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-15T21:44:53.034 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-15T21:44:53.034 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558dffd2bd00) 2024-04-15T21:44:53.034 INFO:journalctl@ceph.mon.b.smithi033.stdout: cache_index_and_filter_blocks: 1 2024-04-15T21:44:53.034 INFO:journalctl@ceph.mon.b.smithi033.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-15T21:44:53.034 INFO:journalctl@ceph.mon.b.smithi033.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-15T21:44:53.034 INFO:journalctl@ceph.mon.b.smithi033.stdout: pin_top_level_index_and_filter: 1 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: index_type: 0 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: data_block_index_type: 0 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: index_shortening: 1 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: hash_index_allow_collision: 1 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: checksum: 1 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: no_block_cache: 0 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_cache: 0x558dffd96170 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_cache_name: BinnedLRUCache 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_cache_options: 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: capacity : 536870912 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: num_shard_bits : 4 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: strict_capacity_limit : 0 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: high_pri_pool_ratio: 0.000 2024-04-15T21:44:53.035 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_cache_compressed: (nil) 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: persistent_cache: (nil) 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_size: 4096 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_size_deviation: 10 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_restart_interval: 16 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: index_block_restart_interval: 1 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: metadata_block_size: 4096 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: partition_filters: 0 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: use_delta_encoding: 1 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: whole_key_filtering: 1 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: verify_compression: 0 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: read_amp_bytes_per_bit: 0 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: format_version: 4 2024-04-15T21:44:53.036 INFO:journalctl@ceph.mon.b.smithi033.stdout: enable_index_compression: 1 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout: block_align: 0 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.write_buffer_size: 33554432 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_write_buffer_number: 2 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compression: NoCompression 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.bottommost_compression: Disabled 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.prefix_extractor: nullptr 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.num_levels: 7 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-15T21:44:53.037 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-15T21:44:53.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-15T21:44:53.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-15T21:44:53.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-15T21:44:53.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-15T21:44:53.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-15T21:44:53.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compression_opts.level: 32767 2024-04-15T21:44:53.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compression_opts.strategy: 0 2024-04-15T21:44:53.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-15T21:44:53.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-15T21:44:53.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-15T21:44:53.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compression_opts.enabled: false 2024-04-15T21:44:53.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-15T21:44:53.038 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.target_file_size_base: 67108864 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-15T21:44:53.039 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.arena_block_size: 4194304 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.disable_auto_compactions: 0 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-15T21:44:53.040 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.table_properties_collectors: 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.inplace_update_support: 0 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.bloom_locality: 0 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.max_successive_merges: 0 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.paranoid_file_checks: 0 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.force_consistency_checks: 1 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.report_bg_io_stats: 0 2024-04-15T21:44:53.041 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.ttl: 2592000 2024-04-15T21:44:53.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-15T21:44:53.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.enable_blob_files: false 2024-04-15T21:44:53.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.min_blob_size: 0 2024-04-15T21:44:53.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.blob_file_size: 268435456 2024-04-15T21:44:53.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-15T21:44:53.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-15T21:44:53.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-15T21:44:53.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-15T21:44:53.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-15T21:44:53.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-15T21:44:53.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713217492599353, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-15T21:44:53.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-15T21:44:53.042 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713217492599651, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1670, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1713217492, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "9c6f2dc9-e655-4176-ba0a-c6f4a3ee14f0", "db_session_id": "J99BV8R4HK00DU7D8AWF"}} 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713217492599957, "job": 1, "event": "recovery_finished"} 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x558dffd7ca80 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: DB pointer 0x558dffd8c000 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** DB Stats ** 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T21:44:53.043 INFO:journalctl@ceph.mon.b.smithi033.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** Compaction Stats [default] ** 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** Compaction Stats [default] ** 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T21:44:53.044 INFO:journalctl@ceph.mon.b.smithi033.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** Compaction Stats [default] ** 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T21:44:53.045 INFO:journalctl@ceph.mon.b.smithi033.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T21:44:53.046 INFO:journalctl@ceph.mon.b.smithi033.stdout: L0 1/0 1.63 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:53.046 INFO:journalctl@ceph.mon.b.smithi033.stdout: Sum 1/0 1.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:53.046 INFO:journalctl@ceph.mon.b.smithi033.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-15T21:44:53.046 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-04-15T21:44:53.046 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** Compaction Stats [default] ** 2024-04-15T21:44:53.046 INFO:journalctl@ceph.mon.b.smithi033.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-15T21:44:53.046 INFO:journalctl@ceph.mon.b.smithi033.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-15T21:44:53.046 INFO:journalctl@ceph.mon.b.smithi033.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.6 0.00 0.00 1 0.000 0 0 2024-04-15T21:44:53.046 INFO:journalctl@ceph.mon.b.smithi033.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-15T21:44:53.046 INFO:journalctl@ceph.mon.b.smithi033.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-15T21:44:53.046 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-15T21:44:53.046 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-15T21:44:53.046 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: using public_addr v2:172.21.15.33:0/0 -> [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: starting mon.b rank -1 at public addrs [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] at bind addrs [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(???) e0 preinit fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing).mds e1 new map 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout: e1 2024-04-15T21:44:53.047 INFO:journalctl@ceph.mon.b.smithi033.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout: legacy client fscid: -1 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout: 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout: No filesystems configured 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T21:44:53.048 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: from='client.? 172.21.15.33:0/2677011753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.a calling monitor election 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.c calling monitor election 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:53.049 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: monmap e2: 2 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: fsmap 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mgrmap e13: y(active, since 35s) 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: overall HEALTH_OK 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: Deploying daemon mon.b on smithi033 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: from='client.? 172.21.15.33:0/451700032' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:53.050 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:53.051 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-15T21:44:53.051 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 ceph-mon[25651]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-15T21:44:53.051 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 bash[25619]: f5d803c8b945f1e57f66466add6390db6777963b7379919e4ff4f3eb0b9aa1a7 2024-04-15T21:44:53.051 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:52 smithi033 systemd[1]: Started Ceph mon.b for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:44:54.317 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-15T21:44:54.318 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph mon dump -f json 2024-04-15T21:44:58.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:58.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:44:58.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: mon.a calling monitor election 2024-04-15T21:44:58.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: mon.c calling monitor election 2024-04-15T21:44:58.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:58.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:58.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:58.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:58.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:58.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: fsmap 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: mgrmap e13: y(active, since 45s) 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: mon.b (rank 2) addr [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] is down (out of quorum) 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:44:58.044 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:44:58.045 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:44:58.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:58.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:44:58.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: mon.a calling monitor election 2024-04-15T21:44:58.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: mon.c calling monitor election 2024-04-15T21:44:58.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:58.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:58.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:58.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:58.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:58.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:58.045 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: fsmap 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: mgrmap e13: y(active, since 45s) 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: mon.b (rank 2) addr [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] is down (out of quorum) 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:44:58.046 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:44:59.200 INFO:teuthology.orchestra.run.smithi033.stdout: 2024-04-15T21:44:59.200 INFO:teuthology.orchestra.run.smithi033.stdout:{"epoch":3,"fsid":"18ed60a0-fb71-11ee-bc8f-c7b262605968","modified":"2024-04-15T21:44:52.619369Z","created":"2024-04-15T21:43:17.375905Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3300","nonce":0},{"type":"v1","addr":"172.21.15.7:6789","nonce":0}]},"addr":"172.21.15.7:6789/0","public_addr":"172.21.15.7:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:3301","nonce":0},{"type":"v1","addr":"172.21.15.7:6790","nonce":0}]},"addr":"172.21.15.7:6790/0","public_addr":"172.21.15.7:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:3300","nonce":0},{"type":"v1","addr":"172.21.15.33:6789","nonce":0}]},"addr":"172.21.15.33:6789/0","public_addr":"172.21.15.33:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-15T21:44:59.202 INFO:teuthology.orchestra.run.smithi033.stderr:dumped monmap epoch 3 2024-04-15T21:44:59.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-15T21:44:59.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:59.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:59.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-15T21:44:59.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: Reconfiguring mon.a (unknown last config time)... 2024-04-15T21:44:59.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T21:44:59.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T21:44:59.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:44:59.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: Reconfiguring daemon mon.a on smithi007 2024-04-15T21:44:59.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[29541]: from='client.? 172.21.15.33:0/3839122414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T21:44:59.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-15T21:44:59.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:59.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:59.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.732 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-15T21:44:59.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: Reconfiguring mon.a (unknown last config time)... 2024-04-15T21:44:59.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T21:44:59.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T21:44:59.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:44:59.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: Reconfiguring daemon mon.a on smithi007 2024-04-15T21:44:59.733 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:44:59 smithi007 ceph-mon[24826]: from='client.? 172.21.15.33:0/3839122414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T21:44:59.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:59.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:44:59.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: mon.a calling monitor election 2024-04-15T21:44:59.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: mon.c calling monitor election 2024-04-15T21:44:59.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:59.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:44:59.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:59.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:59.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: fsmap 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: mgrmap e13: y(active, since 45s) 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: mon.b (rank 2) addr [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] is down (out of quorum) 2024-04-15T21:44:59.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: Reconfiguring mon.a (unknown last config time)... 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T21:44:59.776 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T21:44:59.777 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:44:59.777 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: Reconfiguring daemon mon.a on smithi007 2024-04-15T21:44:59.777 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:44:59 smithi033 ceph-mon[25651]: from='client.? 172.21.15.33:0/3839122414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-15T21:44:59.836 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-15T21:44:59.836 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph config generate-minimal-conf 2024-04-15T21:45:00.863 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: mon.b calling monitor election 2024-04-15T21:45:00.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: mon.b calling monitor election 2024-04-15T21:45:00.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: mon.a calling monitor election 2024-04-15T21:45:00.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: mon.c calling monitor election 2024-04-15T21:45:00.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-15T21:45:00.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-15T21:45:00.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: fsmap 2024-04-15T21:45:00.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T21:45:00.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: mgrmap e13: y(active, since 47s) 2024-04-15T21:45:00.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-15T21:45:00.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: Cluster is now healthy 2024-04-15T21:45:00.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: overall HEALTH_OK 2024-04-15T21:45:00.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:00.864 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: Reconfiguring mon.c (monmap changed)... 2024-04-15T21:45:00.865 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T21:45:00.865 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T21:45:00.865 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:00.865 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[29541]: Reconfiguring daemon mon.c on smithi007 2024-04-15T21:45:00.865 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: mon.b calling monitor election 2024-04-15T21:45:00.865 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: mon.b calling monitor election 2024-04-15T21:45:00.865 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: mon.a calling monitor election 2024-04-15T21:45:00.865 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: mon.c calling monitor election 2024-04-15T21:45:00.865 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-15T21:45:00.865 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-15T21:45:00.865 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: fsmap 2024-04-15T21:45:00.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T21:45:00.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: mgrmap e13: y(active, since 47s) 2024-04-15T21:45:00.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-15T21:45:00.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: Cluster is now healthy 2024-04-15T21:45:00.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: overall HEALTH_OK 2024-04-15T21:45:00.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:00.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: Reconfiguring mon.c (monmap changed)... 2024-04-15T21:45:00.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T21:45:00.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T21:45:00.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:00.866 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:00 smithi007 ceph-mon[24826]: Reconfiguring daemon mon.c on smithi007 2024-04-15T21:45:01.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: mon.b calling monitor election 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: mon.b calling monitor election 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: mon.a calling monitor election 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: mon.c calling monitor election 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: monmap e3: 3 mons at {a=[v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0],b=[v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0],c=[v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0]} 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: fsmap 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: osdmap e4: 0 total, 0 up, 0 in 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: mgrmap e13: y(active, since 47s) 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: Cluster is now healthy 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: overall HEALTH_OK 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: Reconfiguring mon.c (monmap changed)... 2024-04-15T21:45:01.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T21:45:01.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T21:45:01.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:01.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:00 smithi033 ceph-mon[25651]: Reconfiguring daemon mon.c on smithi007 2024-04-15T21:45:01.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:01 smithi007 ceph-mon[24826]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:01.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:01 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:45:01.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:01 smithi007 ceph-mon[29541]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:01.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:01 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:45:02.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:01 smithi033 ceph-mon[25651]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:02.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:01 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:45:02.796 INFO:teuthology.orchestra.run.smithi007.stdout:# minimal ceph.conf for 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:45:02.797 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-04-15T21:45:02.797 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 18ed60a0-fb71-11ee-bc8f-c7b262605968 2024-04-15T21:45:02.797 INFO:teuthology.orchestra.run.smithi007.stdout: mon_host = [v2:172.21.15.7:3300/0,v1:172.21.15.7:6789/0] [v2:172.21.15.33:3300/0,v1:172.21.15.33:6789/0] [v2:172.21.15.7:3301/0,v1:172.21.15.7:6790/0] 2024-04-15T21:45:03.323 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-15T21:45:03.323 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-15T21:45:03.323 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.conf 2024-04-15T21:45:03.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:03.349 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[24826]: Reconfiguring mon.b (monmap changed)... 2024-04-15T21:45:03.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T21:45:03.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T21:45:03.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:03.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[24826]: Reconfiguring daemon mon.b on smithi033 2024-04-15T21:45:03.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:03.350 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/2615238484' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:03.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:03.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[29541]: Reconfiguring mon.b (monmap changed)... 2024-04-15T21:45:03.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T21:45:03.350 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T21:45:03.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:03.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[29541]: Reconfiguring daemon mon.b on smithi033 2024-04-15T21:45:03.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:03.351 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:03 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/2615238484' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:03.354 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-15T21:45:03.354 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T21:45:03.398 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:03 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:03.398 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:03 smithi033 ceph-mon[25651]: Reconfiguring mon.b (monmap changed)... 2024-04-15T21:45:03.398 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:03 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-15T21:45:03.399 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:03 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-15T21:45:03.399 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:03 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:03.399 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:03 smithi033 ceph-mon[25651]: Reconfiguring daemon mon.b on smithi033 2024-04-15T21:45:03.399 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:03 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:03.399 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:03 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/2615238484' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:03.422 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-15T21:45:03.422 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.conf 2024-04-15T21:45:03.454 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-15T21:45:03.454 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-15T21:45:03.523 INFO:tasks.cephadm:Adding mgr.y on smithi007 2024-04-15T21:45:03.523 INFO:tasks.cephadm:Adding mgr.x on smithi033 2024-04-15T21:45:03.523 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch apply mgr '2;smithi007=y;smithi033=x' 2024-04-15T21:45:04.756 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:04 smithi033 ceph-mon[25651]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:04.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:04 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:04.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:04 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:04.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:04 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:04.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:04 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:04.757 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:04 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:04.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:04 smithi007 ceph-mon[24826]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:04.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:04 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:04.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:04 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:04.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:04 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:04.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:04 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:04.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:04 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:04.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:04 smithi007 ceph-mon[29541]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:04.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:04 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:04.795 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:04 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:04.795 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:04 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:04.795 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:04 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:04.795 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:04 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:05.250 INFO:teuthology.orchestra.run.smithi033.stdout:Scheduled mgr update... 2024-04-15T21:45:05.882 DEBUG:teuthology.orchestra.run.smithi033:mgr.x> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mgr.x.service 2024-04-15T21:45:05.885 INFO:tasks.cephadm:Deploying OSDs... 2024-04-15T21:45:05.885 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-15T21:45:05.885 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T21:45:05.904 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T21:45:05.905 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-04-15T21:45:05.962 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T21:45:05.962 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T21:45:05.963 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-15T21:45:05.963 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T21:45:05.963 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T21:45:05.963 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-04-15 21:44:42.207118668 +0000 2024-04-15T21:45:05.963 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-04-15 21:42:19.155749523 +0000 2024-04-15T21:45:05.963 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-04-15 21:42:19.155749523 +0000 2024-04-15T21:45:05.963 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-04-15 21:42:19.155749523 +0000 2024-04-15T21:45:05.963 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T21:45:06.032 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-04-15T21:45:06.032 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-04-15T21:45:06.032 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000117613 s, 4.4 MB/s 2024-04-15T21:45:06.033 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T21:45:06.092 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-04-15T21:45:06.149 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T21:45:06.150 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T21:45:06.150 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-15T21:45:06.150 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T21:45:06.150 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T21:45:06.150 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-04-15 21:44:42.207118668 +0000 2024-04-15T21:45:06.150 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-04-15 21:42:19.510744517 +0000 2024-04-15T21:45:06.150 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-04-15 21:42:19.510744517 +0000 2024-04-15T21:45:06.150 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-04-15 21:42:19.510744517 +0000 2024-04-15T21:45:06.151 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T21:45:06.220 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-04-15T21:45:06.220 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-04-15T21:45:06.220 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000117878 s, 4.3 MB/s 2024-04-15T21:45:06.221 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T21:45:06.280 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-04-15T21:45:06.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[24826]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:06.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[24826]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi033=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:45:06.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[24826]: Saving service mgr spec with placement smithi007=y;smithi033=x;count:2 2024-04-15T21:45:06.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:06.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:06.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:06.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:06.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:06.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T21:45:06.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-15T21:45:06.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T21:45:06.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:06.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[24826]: Deploying daemon mgr.x on smithi033 2024-04-15T21:45:06.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[29541]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:06.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[29541]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi033=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:45:06.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[29541]: Saving service mgr spec with placement smithi007=y;smithi033=x;count:2 2024-04-15T21:45:06.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:06.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:06.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:06.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:06.295 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:06.295 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T21:45:06.295 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-15T21:45:06.295 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T21:45:06.295 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:06.295 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:06 smithi007 ceph-mon[29541]: Deploying daemon mgr.x on smithi033 2024-04-15T21:45:06.311 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T21:45:06.311 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T21:45:06.311 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-15T21:45:06.311 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T21:45:06.311 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T21:45:06.311 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-04-15 21:44:42.207118668 +0000 2024-04-15T21:45:06.312 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-04-15 21:42:19.853739679 +0000 2024-04-15T21:45:06.312 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-04-15 21:42:19.853739679 +0000 2024-04-15T21:45:06.312 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-04-15 21:42:19.853739679 +0000 2024-04-15T21:45:06.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T21:45:06.380 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-04-15T21:45:06.380 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-04-15T21:45:06.380 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000113756 s, 4.5 MB/s 2024-04-15T21:45:06.381 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T21:45:06.439 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-04-15T21:45:06.446 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:06 smithi033 ceph-mon[25651]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:06.446 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:06 smithi033 ceph-mon[25651]: from='client.24110 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi007=y;smithi033=x", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:45:06.446 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:06 smithi033 ceph-mon[25651]: Saving service mgr spec with placement smithi007=y;smithi033=x;count:2 2024-04-15T21:45:06.446 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:06.446 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:06.446 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:06.446 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:06.446 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:06.446 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T21:45:06.446 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-15T21:45:06.446 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T21:45:06.447 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:06.447 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:06 smithi033 ceph-mon[25651]: Deploying daemon mgr.x on smithi033 2024-04-15T21:45:06.496 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T21:45:06.496 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T21:45:06.496 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-15T21:45:06.496 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T21:45:06.496 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T21:45:06.496 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-04-15 21:44:42.208118654 +0000 2024-04-15T21:45:06.496 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-04-15 21:42:20.190734927 +0000 2024-04-15T21:45:06.496 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-04-15 21:42:20.190734927 +0000 2024-04-15T21:45:06.496 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-04-15 21:42:20.190734927 +0000 2024-04-15T21:45:06.497 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T21:45:06.565 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-04-15T21:45:06.565 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-04-15T21:45:06.565 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.000103422 s, 5.0 MB/s 2024-04-15T21:45:06.566 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T21:45:06.624 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-15T21:45:06.624 DEBUG:teuthology.orchestra.run.smithi033:> dd if=/scratch_devs of=/dev/stdout 2024-04-15T21:45:06.641 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-15T21:45:06.641 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_1 2024-04-15T21:45:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-15T21:45:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T21:45:06.698 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-15T21:45:06.698 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T21:45:06.698 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T21:45:06.698 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-04-15 21:44:51.411938181 +0000 2024-04-15T21:45:06.698 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-04-15 21:42:21.346045267 +0000 2024-04-15T21:45:06.698 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-04-15 21:42:21.346045267 +0000 2024-04-15T21:45:06.698 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-04-15 21:42:21.346045267 +0000 2024-04-15T21:45:06.699 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-15T21:45:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-04-15T21:45:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-04-15T21:45:06.768 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000248053 s, 2.1 MB/s 2024-04-15T21:45:06.769 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-15T21:45:06.826 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_2 2024-04-15T21:45:06.882 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-15T21:45:06.883 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T21:45:06.883 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-15T21:45:06.883 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T21:45:06.883 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T21:45:06.883 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-04-15 21:44:51.412938164 +0000 2024-04-15T21:45:06.883 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-04-15 21:42:21.700039294 +0000 2024-04-15T21:45:06.883 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-04-15 21:42:21.700039294 +0000 2024-04-15T21:45:06.883 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-04-15 21:42:21.700039294 +0000 2024-04-15T21:45:06.883 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-15T21:45:06.950 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-04-15T21:45:06.951 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-04-15T21:45:06.951 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000208358 s, 2.5 MB/s 2024-04-15T21:45:06.952 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-15T21:45:07.008 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_3 2024-04-15T21:45:07.063 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-15T21:45:07.064 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T21:45:07.064 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-15T21:45:07.064 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T21:45:07.064 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T21:45:07.064 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-04-15 21:44:51.412938164 +0000 2024-04-15T21:45:07.064 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-04-15 21:42:22.052033355 +0000 2024-04-15T21:45:07.064 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-04-15 21:42:22.052033355 +0000 2024-04-15T21:45:07.064 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-04-15 21:42:22.052033355 +0000 2024-04-15T21:45:07.064 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-15T21:45:07.267 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-04-15T21:45:07.268 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-04-15T21:45:07.268 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.0002096 s, 2.4 MB/s 2024-04-15T21:45:07.269 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-15T21:45:07.285 DEBUG:teuthology.orchestra.run.smithi033:> stat /dev/vg_nvme/lv_4 2024-04-15T21:45:07.345 INFO:teuthology.orchestra.run.smithi033.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-15T21:45:07.346 INFO:teuthology.orchestra.run.smithi033.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-15T21:45:07.346 INFO:teuthology.orchestra.run.smithi033.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-04-15T21:45:07.346 INFO:teuthology.orchestra.run.smithi033.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-15T21:45:07.346 INFO:teuthology.orchestra.run.smithi033.stdout:Context: system_u:object_r:device_t:s0 2024-04-15T21:45:07.346 INFO:teuthology.orchestra.run.smithi033.stdout:Access: 2024-04-15 21:45:07.223674282 +0000 2024-04-15T21:45:07.346 INFO:teuthology.orchestra.run.smithi033.stdout:Modify: 2024-04-15 21:42:22.400027484 +0000 2024-04-15T21:45:07.346 INFO:teuthology.orchestra.run.smithi033.stdout:Change: 2024-04-15 21:42:22.400027484 +0000 2024-04-15T21:45:07.346 INFO:teuthology.orchestra.run.smithi033.stdout: Birth: 2024-04-15 21:42:22.400027484 +0000 2024-04-15T21:45:07.346 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-15T21:45:07.444 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records in 2024-04-15T21:45:07.444 INFO:teuthology.orchestra.run.smithi033.stderr:1+0 records out 2024-04-15T21:45:07.444 INFO:teuthology.orchestra.run.smithi033.stderr:512 bytes copied, 0.000192357 s, 2.7 MB/s 2024-04-15T21:45:07.445 DEBUG:teuthology.orchestra.run.smithi033:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-15T21:45:07.461 INFO:tasks.cephadm:Deploying osd.0 on smithi007 with /dev/vg_nvme/lv_4... 2024-04-15T21:45:07.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-15T21:45:07.675 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:07 smithi033 systemd[1]: Starting Ceph mgr.x for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:45:07.978 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:07 smithi033 podman[26814]: 2024-04-15 21:45:07.577045569 +0000 UTC m=+0.018665612 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T21:45:07.979 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:07 smithi033 podman[26814]: 2024-04-15 21:45:07.682799333 +0000 UTC m=+0.124419376 container create 6202e275d3623c349623fdb6c5bb2c30e572497bdf5460edbbd5ae094505e962 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x, release=754, distribution-scope=public, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, name=centos-stream, vcs-type=git, ceph=True, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-15T21:45:07.979 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:07 smithi033 podman[26814]: 2024-04-15 21:45:07.876775745 +0000 UTC m=+0.318395788 container init 6202e275d3623c349623fdb6c5bb2c30e572497bdf5460edbbd5ae094505e962 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, release=754, name=centos-stream, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=HEAD, com.redhat.component=centos-stream-container, distribution-scope=public, architecture=x86_64) 2024-04-15T21:45:07.979 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:07 smithi033 podman[26814]: 2024-04-15 21:45:07.879528703 +0000 UTC m=+0.321148746 container start 6202e275d3623c349623fdb6c5bb2c30e572497bdf5460edbbd5ae094505e962 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, vendor=Red Hat, Inc., RELEASE=HEAD, vcs-type=git, ceph=True, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, name=centos-stream, io.k8s.display-name=CentOS Stream 8, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0) 2024-04-15T21:45:07.979 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:07 smithi033 ceph-mgr[26834]: -- 172.21.15.33:0/2461675435 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5609f3f5a340 con 0x5609f3f33000 2024-04-15T21:45:07.979 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:07 smithi033 bash[26814]: 6202e275d3623c349623fdb6c5bb2c30e572497bdf5460edbbd5ae094505e962 2024-04-15T21:45:08.255 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:07 smithi033 systemd[1]: Started Ceph mgr.x for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:45:08.523 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:08.401+0000 7f6b7b3ce000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T21:45:08.523 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:08.501+0000 7f6b7b3ce000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T21:45:08.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:08 smithi033 ceph-mon[25651]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:08.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:08 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:08.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:08 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:08.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:08 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:08.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:08 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:08.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:08 smithi007 ceph-mon[24826]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:08.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:08 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:08.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:08 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:08.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:08 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:08.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:08 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:08.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:08 smithi007 ceph-mon[29541]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:08.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:08 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:08.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:08 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:08.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:08 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:08.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:08 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:09.309 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:09.130+0000 7f6b7b3ce000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T21:45:09.310 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:09.221+0000 7f6b7b3ce000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T21:45:09.310 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:09.306+0000 7f6b7b3ce000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T21:45:09.570 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-15T21:45:09.571 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-15T21:45:09.571 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-04-15T21:45:09.571 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-04-15T21:45:09.571 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00849811 s, 1.2 GB/s 2024-04-15T21:45:09.571 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-04-15T21:45:09.656 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:09.458+0000 7f6b7b3ce000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T21:45:09.656 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:09.541+0000 7f6b7b3ce000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T21:45:09.656 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:09.653+0000 7f6b7b3ce000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T21:45:09.906 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:09.738+0000 7f6b7b3ce000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T21:45:09.906 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:09.903+0000 7f6b7b3ce000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T21:45:10.200 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[29541]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:10.201 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:10.201 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:10.201 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[29541]: Reconfiguring mgr.y (unknown last config time)... 2024-04-15T21:45:10.201 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T21:45:10.201 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T21:45:10.201 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:10.201 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[29541]: Reconfiguring daemon mgr.y on smithi007 2024-04-15T21:45:10.201 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[24826]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:10.201 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:10.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:10.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[24826]: Reconfiguring mgr.y (unknown last config time)... 2024-04-15T21:45:10.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T21:45:10.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T21:45:10.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:10.202 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:10 smithi007 ceph-mon[24826]: Reconfiguring daemon mgr.y on smithi007 2024-04-15T21:45:10.273 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:09.990+0000 7f6b7b3ce000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T21:45:10.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:10 smithi033 ceph-mon[25651]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:10.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:10 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:10.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:10 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:10.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:10 smithi033 ceph-mon[25651]: Reconfiguring mgr.y (unknown last config time)... 2024-04-15T21:45:10.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:10 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-15T21:45:10.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:10 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-15T21:45:10.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:10 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:10.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:10 smithi033 ceph-mon[25651]: Reconfiguring daemon mgr.y on smithi007 2024-04-15T21:45:10.938 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_4 2024-04-15T21:45:11.390 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:11 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:11.059+0000 7f6b7b3ce000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T21:45:11.390 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:11 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:11.192+0000 7f6b7b3ce000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T21:45:11.390 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:11 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:11.387+0000 7f6b7b3ce000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T21:45:11.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:11 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:11.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:11 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:11.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:11 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:11.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:11 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:11.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:11 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:11.774 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:11 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:11.475+0000 7f6b7b3ce000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T21:45:11.774 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:11 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:11.615+0000 7f6b7b3ce000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T21:45:11.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:11 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:11.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:11 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:11.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:11 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:11.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:11 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:11.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:11 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:11.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:11 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:11.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:11 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:11.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:11 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:11.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:11 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:11.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:11 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:12.254 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:11 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:11.938+0000 7f6b7b3ce000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T21:45:12.254 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:12 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:12.165+0000 7f6b7b3ce000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T21:45:12.255 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:12 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:12.251+0000 7f6b7b3ce000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T21:45:12.523 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:12 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:12.427+0000 7f6b7b3ce000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T21:45:12.788 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:12 smithi007 ceph-mon[24826]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:12.788 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:12 smithi007 ceph-mon[29541]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:13.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:12 smithi033 ceph-mon[25651]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:13.676 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:45:13.677 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:45:13.677 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:13.677 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:45:13.677 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:45:13.677 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:13.677 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:13 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:13.370+0000 7f6b7b3ce000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T21:45:13.677 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:13 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:13.674+0000 7f6b7b3ce000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T21:45:13.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:45:13.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:45:13.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:13.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:45:13.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:45:13.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:13.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:45:13.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:45:13.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:13.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:45:13.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:45:13.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:13.927 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:13 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:13.770+0000 7f6b7b3ce000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T21:45:13.927 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:45:13 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:45:13.924+0000 7f6b7b3ce000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T21:45:15.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:14 smithi033 ceph-mon[25651]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:15.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:14 smithi033 ceph-mon[25651]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:45:15.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:14 smithi033 ceph-mon[25651]: Standby manager daemon x started 2024-04-15T21:45:15.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:14 smithi033 ceph-mon[25651]: from='mgr.? 172.21.15.33:0/1961804320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T21:45:15.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:14 smithi033 ceph-mon[25651]: from='mgr.? 172.21.15.33:0/1961804320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T21:45:15.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:14 smithi033 ceph-mon[25651]: from='mgr.? 172.21.15.33:0/1961804320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T21:45:15.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:14 smithi033 ceph-mon[25651]: from='mgr.? 172.21.15.33:0/1961804320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T21:45:15.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[24826]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:15.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[24826]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:45:15.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[24826]: Standby manager daemon x started 2024-04-15T21:45:15.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[24826]: from='mgr.? 172.21.15.33:0/1961804320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T21:45:15.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[24826]: from='mgr.? 172.21.15.33:0/1961804320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T21:45:15.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[24826]: from='mgr.? 172.21.15.33:0/1961804320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T21:45:15.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[24826]: from='mgr.? 172.21.15.33:0/1961804320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T21:45:15.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[29541]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:15.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[29541]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:45:15.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[29541]: Standby manager daemon x started 2024-04-15T21:45:15.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[29541]: from='mgr.? 172.21.15.33:0/1961804320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T21:45:15.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[29541]: from='mgr.? 172.21.15.33:0/1961804320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T21:45:15.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[29541]: from='mgr.? 172.21.15.33:0/1961804320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T21:45:15.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:14 smithi007 ceph-mon[29541]: from='mgr.? 172.21.15.33:0/1961804320' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T21:45:16.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:15 smithi033 ceph-mon[25651]: mgrmap e14: y(active, since 62s), standbys: x 2024-04-15T21:45:16.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:15 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T21:45:16.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:15 smithi033 ceph-mon[25651]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:16.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:15 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/177860305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57b79fc0-6f79-4eec-9331-1a2f7f45d0f1"}]: dispatch 2024-04-15T21:45:16.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:15 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/177860305' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57b79fc0-6f79-4eec-9331-1a2f7f45d0f1"}]': finished 2024-04-15T21:45:16.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:15 smithi033 ceph-mon[25651]: osdmap e5: 1 total, 0 up, 1 in 2024-04-15T21:45:16.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:15 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:16.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:15 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1761641160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:45:16.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[24826]: mgrmap e14: y(active, since 62s), standbys: x 2024-04-15T21:45:16.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T21:45:16.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[24826]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:16.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/177860305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57b79fc0-6f79-4eec-9331-1a2f7f45d0f1"}]: dispatch 2024-04-15T21:45:16.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/177860305' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57b79fc0-6f79-4eec-9331-1a2f7f45d0f1"}]': finished 2024-04-15T21:45:16.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[24826]: osdmap e5: 1 total, 0 up, 1 in 2024-04-15T21:45:16.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:16.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1761641160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:45:16.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[29541]: mgrmap e14: y(active, since 62s), standbys: x 2024-04-15T21:45:16.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T21:45:16.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[29541]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:16.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/177860305' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57b79fc0-6f79-4eec-9331-1a2f7f45d0f1"}]: dispatch 2024-04-15T21:45:16.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/177860305' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57b79fc0-6f79-4eec-9331-1a2f7f45d0f1"}]': finished 2024-04-15T21:45:16.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[29541]: osdmap e5: 1 total, 0 up, 1 in 2024-04-15T21:45:16.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:16.044 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:15 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1761641160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:45:17.925 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:17 smithi007 ceph-mon[24826]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:17.926 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:17 smithi007 ceph-mon[29541]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:18.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:17 smithi033 ceph-mon[25651]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:20.373 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:20 smithi007 ceph-mon[24826]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:20.373 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:20 smithi007 ceph-mon[29541]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:20.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:20 smithi033 ceph-mon[25651]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:21.483 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:21 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-15T21:45:21.627 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:21 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:21.627 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:21 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-15T21:45:21.627 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:21 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:21.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:21 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-15T21:45:21.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:21 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:22.674 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:22 smithi033 ceph-mon[25651]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:22.674 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:22 smithi033 ceph-mon[25651]: Deploying daemon osd.0 on smithi007 2024-04-15T21:45:22.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:22 smithi007 ceph-mon[29541]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:22.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:22 smithi007 ceph-mon[29541]: Deploying daemon osd.0 on smithi007 2024-04-15T21:45:22.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:22 smithi007 ceph-mon[24826]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:22.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:22 smithi007 ceph-mon[24826]: Deploying daemon osd.0 on smithi007 2024-04-15T21:45:24.657 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:24 smithi007 ceph-mon[24826]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:24.658 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:24 smithi007 ceph-mon[29541]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:24.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:24 smithi033 ceph-mon[25651]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:26.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:26 smithi007 ceph-mon[24826]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:26.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:26 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:26.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:26 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:26.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:26 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:26.743 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:26 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:26.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:26 smithi007 ceph-mon[29541]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:26.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:26 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:26.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:26 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:26.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:26 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:26.744 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:26 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:26.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:26 smithi033 ceph-mon[25651]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:26.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:26 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:26.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:26 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:26.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:26 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:26.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:26 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:28.618 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:28 smithi007 ceph-mon[29541]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:28.619 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:28 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:28.619 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:28 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:28.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:28 smithi007 ceph-mon[24826]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:28.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:28 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:28.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:28 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:28.649 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 0 on host 'smithi007' 2024-04-15T21:45:28.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:28 smithi033 ceph-mon[25651]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:28.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:28 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:28.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:28 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:29.236 DEBUG:teuthology.orchestra.run.smithi007:osd.0> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.0.service 2024-04-15T21:45:29.239 INFO:tasks.cephadm:Deploying osd.1 on smithi007 with /dev/vg_nvme/lv_3... 2024-04-15T21:45:29.239 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-15T21:45:29.495 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:29 smithi007 ceph-mon[24826]: from='osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-15T21:45:29.495 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:29 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:29.495 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:29 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:29.495 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:29 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:29.495 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:29 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:29.495 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:29 smithi007 ceph-mon[29541]: from='osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-15T21:45:29.495 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:29 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:29.495 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:29 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:29.495 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:29 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:29.495 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:29 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:29.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:29 smithi033 ceph-mon[25651]: from='osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-15T21:45:29.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:29 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:29.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:29 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:29.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:29 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:29.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:29 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:30.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:30 smithi007 ceph-mon[24826]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:30.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:30 smithi007 ceph-mon[24826]: from='osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-15T21:45:30.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:30 smithi007 ceph-mon[24826]: osdmap e6: 1 total, 0 up, 1 in 2024-04-15T21:45:30.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:30 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:30.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:30 smithi007 ceph-mon[24826]: from='osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:45:30.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:30 smithi007 ceph-mon[29541]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:30.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:30 smithi007 ceph-mon[29541]: from='osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-15T21:45:30.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:30 smithi007 ceph-mon[29541]: osdmap e6: 1 total, 0 up, 1 in 2024-04-15T21:45:30.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:30 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:30.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:30 smithi007 ceph-mon[29541]: from='osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:45:30.544 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:45:30 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0[32406]: 2024-04-15T21:45:30.444+0000 7fd06fdd9700 -1 osd.0 0 waiting for initial osdmap 2024-04-15T21:45:30.544 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:45:30 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0[32406]: 2024-04-15T21:45:30.447+0000 7fd06c774700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T21:45:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:30 smithi033 ceph-mon[25651]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:30 smithi033 ceph-mon[25651]: from='osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-15T21:45:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:30 smithi033 ceph-mon[25651]: osdmap e6: 1 total, 0 up, 1 in 2024-04-15T21:45:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:30 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:30 smithi033 ceph-mon[25651]: from='osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:45:31.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:31 smithi007 ceph-mon[24826]: from='osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-15T21:45:31.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:31 smithi007 ceph-mon[24826]: osdmap e7: 1 total, 0 up, 1 in 2024-04-15T21:45:31.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:31 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:31.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:31 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:31.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:31 smithi007 ceph-mon[29541]: from='osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-15T21:45:31.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:31 smithi007 ceph-mon[29541]: osdmap e7: 1 total, 0 up, 1 in 2024-04-15T21:45:31.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:31 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:31.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:31 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:31.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:31 smithi033 ceph-mon[25651]: from='osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-15T21:45:31.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:31 smithi033 ceph-mon[25651]: osdmap e7: 1 total, 0 up, 1 in 2024-04-15T21:45:31.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:31 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:31.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:31 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-15T21:45:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-15T21:45:32.028 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-04-15T21:45:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-04-15T21:45:32.028 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00950804 s, 1.1 GB/s 2024-04-15T21:45:32.028 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-04-15T21:45:32.651 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_3 2024-04-15T21:45:32.677 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:32 smithi007 ceph-mon[24826]: purged_snaps scrub starts 2024-04-15T21:45:32.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:32 smithi007 ceph-mon[24826]: purged_snaps scrub ok 2024-04-15T21:45:32.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:32 smithi007 ceph-mon[24826]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:32.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:32 smithi007 ceph-mon[24826]: osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297] boot 2024-04-15T21:45:32.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:32 smithi007 ceph-mon[24826]: osdmap e8: 1 total, 1 up, 1 in 2024-04-15T21:45:32.679 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:32 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:32.679 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:32 smithi007 ceph-mon[29541]: purged_snaps scrub starts 2024-04-15T21:45:32.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:32 smithi007 ceph-mon[29541]: purged_snaps scrub ok 2024-04-15T21:45:32.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:32 smithi007 ceph-mon[29541]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:32.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:32 smithi007 ceph-mon[29541]: osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297] boot 2024-04-15T21:45:32.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:32 smithi007 ceph-mon[29541]: osdmap e8: 1 total, 1 up, 1 in 2024-04-15T21:45:32.680 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:32 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:32 smithi033 ceph-mon[25651]: purged_snaps scrub starts 2024-04-15T21:45:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:32 smithi033 ceph-mon[25651]: purged_snaps scrub ok 2024-04-15T21:45:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:32 smithi033 ceph-mon[25651]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-15T21:45:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:32 smithi033 ceph-mon[25651]: osd.0 [v2:172.21.15.7:6802/2067680297,v1:172.21.15.7:6803/2067680297] boot 2024-04-15T21:45:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:32 smithi033 ceph-mon[25651]: osdmap e8: 1 total, 1 up, 1 in 2024-04-15T21:45:32.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:32 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:45:33.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:33 smithi033 ceph-mon[25651]: osdmap e9: 1 total, 1 up, 1 in 2024-04-15T21:45:33.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:33 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:33.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:33 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:45:33.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:33 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:33.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:33 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:33.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:33 smithi007 ceph-mon[24826]: osdmap e9: 1 total, 1 up, 1 in 2024-04-15T21:45:33.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:33 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:33.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:33 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:45:33.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:33 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:33.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:33 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:33.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:33 smithi007 ceph-mon[29541]: osdmap e9: 1 total, 1 up, 1 in 2024-04-15T21:45:33.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:33 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:33.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:33 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:45:33.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:33 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:33.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:33 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:34.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:34 smithi033 ceph-mon[25651]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:34.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:34 smithi033 ceph-mon[25651]: Detected new or changed devices on smithi007 2024-04-15T21:45:34.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:34 smithi033 ceph-mon[25651]: Adjusting osd_memory_target on smithi007 to 16069M 2024-04-15T21:45:34.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:34 smithi007 ceph-mon[24826]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:34.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:34 smithi007 ceph-mon[24826]: Detected new or changed devices on smithi007 2024-04-15T21:45:34.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:34 smithi007 ceph-mon[24826]: Adjusting osd_memory_target on smithi007 to 16069M 2024-04-15T21:45:34.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:34 smithi007 ceph-mon[29541]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:34.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:34 smithi007 ceph-mon[29541]: Detected new or changed devices on smithi007 2024-04-15T21:45:34.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:34 smithi007 ceph-mon[29541]: Adjusting osd_memory_target on smithi007 to 16069M 2024-04-15T21:45:35.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:35 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:45:35.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:35 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:45:35.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:35 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:35.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:35 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:45:35.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:35 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:45:35.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:35 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:35.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:35 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:45:35.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:35 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:45:35.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:35 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:36.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:36 smithi033 ceph-mon[25651]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:36.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:36 smithi033 ceph-mon[25651]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:45:36.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:36 smithi007 ceph-mon[24826]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:36.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:36 smithi007 ceph-mon[24826]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:45:36.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:36 smithi007 ceph-mon[29541]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:36.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:36 smithi007 ceph-mon[29541]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:45:37.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:37 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3804255677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9184457-c172-4ea2-a429-49cd5ce3a01a"}]: dispatch 2024-04-15T21:45:37.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:37 smithi033 ceph-mon[25651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9184457-c172-4ea2-a429-49cd5ce3a01a"}]: dispatch 2024-04-15T21:45:37.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:37 smithi033 ceph-mon[25651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9184457-c172-4ea2-a429-49cd5ce3a01a"}]': finished 2024-04-15T21:45:37.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:37 smithi033 ceph-mon[25651]: osdmap e10: 2 total, 1 up, 2 in 2024-04-15T21:45:37.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:37 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:37.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:37 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3804255677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9184457-c172-4ea2-a429-49cd5ce3a01a"}]: dispatch 2024-04-15T21:45:37.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:37 smithi007 ceph-mon[24826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9184457-c172-4ea2-a429-49cd5ce3a01a"}]: dispatch 2024-04-15T21:45:37.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:37 smithi007 ceph-mon[24826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9184457-c172-4ea2-a429-49cd5ce3a01a"}]': finished 2024-04-15T21:45:37.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:37 smithi007 ceph-mon[24826]: osdmap e10: 2 total, 1 up, 2 in 2024-04-15T21:45:37.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:37 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:37.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:37 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3804255677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9184457-c172-4ea2-a429-49cd5ce3a01a"}]: dispatch 2024-04-15T21:45:37.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:37 smithi007 ceph-mon[29541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e9184457-c172-4ea2-a429-49cd5ce3a01a"}]: dispatch 2024-04-15T21:45:37.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:37 smithi007 ceph-mon[29541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e9184457-c172-4ea2-a429-49cd5ce3a01a"}]': finished 2024-04-15T21:45:37.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:37 smithi007 ceph-mon[29541]: osdmap e10: 2 total, 1 up, 2 in 2024-04-15T21:45:37.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:37 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:38.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:38 smithi033 ceph-mon[25651]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:38.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:38 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3128916303' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:45:38.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:38 smithi007 ceph-mon[24826]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:38.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:38 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3128916303' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:45:38.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:38 smithi007 ceph-mon[29541]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:38.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:38 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3128916303' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:45:40.480 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:40 smithi007 ceph-mon[24826]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:40.481 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:40 smithi007 ceph-mon[29541]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:40.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:40 smithi033 ceph-mon[25651]: pgmap v43: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:42.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:42 smithi033 ceph-mon[25651]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:42.773 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:42 smithi007 ceph-mon[24826]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:42.774 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:42 smithi007 ceph-mon[29541]: pgmap v44: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:43.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:43 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-15T21:45:43.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:43 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:43.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:43 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-15T21:45:43.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:43 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:43.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:43 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-15T21:45:43.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:43 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:44.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:44 smithi007 ceph-mon[24826]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:44.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:44 smithi007 ceph-mon[24826]: Deploying daemon osd.1 on smithi007 2024-04-15T21:45:44.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:44 smithi007 ceph-mon[29541]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:44.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:44 smithi007 ceph-mon[29541]: Deploying daemon osd.1 on smithi007 2024-04-15T21:45:44.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:44 smithi033 ceph-mon[25651]: pgmap v45: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:44.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:44 smithi033 ceph-mon[25651]: Deploying daemon osd.1 on smithi007 2024-04-15T21:45:46.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:46 smithi033 ceph-mon[25651]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:46.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:46 smithi007 ceph-mon[24826]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:46.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:46 smithi007 ceph-mon[29541]: pgmap v46: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:48.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:48 smithi033 ceph-mon[25651]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:48.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:48 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:48.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:48 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:48.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:48 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:48.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:48 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:48.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:48 smithi007 ceph-mon[29541]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:48.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:48 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:48.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:48 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:48.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:48 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:48.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:48 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:48.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:48 smithi007 ceph-mon[24826]: pgmap v47: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:48.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:48 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:48.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:48 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:48.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:48 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:48.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:48 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:50.668 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 1 on host 'smithi007' 2024-04-15T21:45:50.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:50 smithi033 ceph-mon[25651]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:50.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:50 smithi007 ceph-mon[24826]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:50.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:50 smithi007 ceph-mon[29541]: pgmap v48: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:51.292 DEBUG:teuthology.orchestra.run.smithi007:osd.1> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.1.service 2024-04-15T21:45:51.335 INFO:tasks.cephadm:Deploying osd.2 on smithi007 with /dev/vg_nvme/lv_2... 2024-04-15T21:45:51.335 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-15T21:45:51.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:51 smithi007 ceph-mon[24826]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:51.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:51 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:51.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:51 smithi007 ceph-mon[24826]: from='osd.1 [v2:172.21.15.7:6810/1851194794,v1:172.21.15.7:6811/1851194794]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T21:45:51.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:51 smithi007 ceph-mon[24826]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T21:45:51.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:51 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:51.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:51 smithi007 ceph-mon[29541]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:51.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:51 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:51.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:51 smithi007 ceph-mon[29541]: from='osd.1 [v2:172.21.15.7:6810/1851194794,v1:172.21.15.7:6811/1851194794]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T21:45:51.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:51 smithi007 ceph-mon[29541]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T21:45:51.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:51 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:52.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:51 smithi033 ceph-mon[25651]: pgmap v49: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:52.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:51 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:52.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:51 smithi033 ceph-mon[25651]: from='osd.1 [v2:172.21.15.7:6810/1851194794,v1:172.21.15.7:6811/1851194794]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T21:45:52.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:51 smithi033 ceph-mon[25651]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-15T21:45:52.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:51 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:52.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:52 smithi007 ceph-mon[29541]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-15T21:45:52.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:52 smithi007 ceph-mon[29541]: osdmap e11: 2 total, 1 up, 2 in 2024-04-15T21:45:52.937 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:52 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:52.938 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:52 smithi007 ceph-mon[29541]: from='osd.1 [v2:172.21.15.7:6810/1851194794,v1:172.21.15.7:6811/1851194794]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:45:52.938 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:52 smithi007 ceph-mon[29541]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:45:52.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:52 smithi007 ceph-mon[24826]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-15T21:45:52.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:52 smithi007 ceph-mon[24826]: osdmap e11: 2 total, 1 up, 2 in 2024-04-15T21:45:52.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:52 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:52.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:52 smithi007 ceph-mon[24826]: from='osd.1 [v2:172.21.15.7:6810/1851194794,v1:172.21.15.7:6811/1851194794]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:45:52.938 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:52 smithi007 ceph-mon[24826]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:45:52.939 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:45:52 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1[35375]: 2024-04-15T21:45:52.684+0000 7f51347f8700 -1 osd.1 0 waiting for initial osdmap 2024-04-15T21:45:52.939 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:45:52 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1[35375]: 2024-04-15T21:45:52.687+0000 7f512e98e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T21:45:53.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:52 smithi033 ceph-mon[25651]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-15T21:45:53.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:52 smithi033 ceph-mon[25651]: osdmap e11: 2 total, 1 up, 2 in 2024-04-15T21:45:53.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:53.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:52 smithi033 ceph-mon[25651]: from='osd.1 [v2:172.21.15.7:6810/1851194794,v1:172.21.15.7:6811/1851194794]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:45:53.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:52 smithi033 ceph-mon[25651]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:45:53.227 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-15T21:45:53.228 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-15T21:45:53.228 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-04-15T21:45:53.228 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-04-15T21:45:53.228 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110726 s, 947 MB/s 2024-04-15T21:45:53.228 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-04-15T21:45:53.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:53 smithi033 ceph-mon[25651]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:53.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:53 smithi033 ceph-mon[25651]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-15T21:45:53.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:53 smithi033 ceph-mon[25651]: osdmap e12: 2 total, 1 up, 2 in 2024-04-15T21:45:53.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:53 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:53.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:53 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:53.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:53 smithi007 ceph-mon[24826]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:53.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:53 smithi007 ceph-mon[24826]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-15T21:45:53.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:53 smithi007 ceph-mon[24826]: osdmap e12: 2 total, 1 up, 2 in 2024-04-15T21:45:53.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:53 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:53.993 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:53 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:53.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:53 smithi007 ceph-mon[29541]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-15T21:45:53.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:53 smithi007 ceph-mon[29541]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-15T21:45:53.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:53 smithi007 ceph-mon[29541]: osdmap e12: 2 total, 1 up, 2 in 2024-04-15T21:45:53.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:53 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:53.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:53 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:54.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:54 smithi007 ceph-mon[24826]: purged_snaps scrub starts 2024-04-15T21:45:54.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:54 smithi007 ceph-mon[24826]: purged_snaps scrub ok 2024-04-15T21:45:54.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:54 smithi007 ceph-mon[24826]: osd.1 [v2:172.21.15.7:6810/1851194794,v1:172.21.15.7:6811/1851194794] boot 2024-04-15T21:45:54.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:54 smithi007 ceph-mon[24826]: osdmap e13: 2 total, 2 up, 2 in 2024-04-15T21:45:54.950 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:54 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:54.950 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:54 smithi007 ceph-mon[29541]: purged_snaps scrub starts 2024-04-15T21:45:54.950 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:54 smithi007 ceph-mon[29541]: purged_snaps scrub ok 2024-04-15T21:45:54.950 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:54 smithi007 ceph-mon[29541]: osd.1 [v2:172.21.15.7:6810/1851194794,v1:172.21.15.7:6811/1851194794] boot 2024-04-15T21:45:54.950 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:54 smithi007 ceph-mon[29541]: osdmap e13: 2 total, 2 up, 2 in 2024-04-15T21:45:54.951 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:54 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:54.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_2 2024-04-15T21:45:55.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:54 smithi033 ceph-mon[25651]: purged_snaps scrub starts 2024-04-15T21:45:55.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:54 smithi033 ceph-mon[25651]: purged_snaps scrub ok 2024-04-15T21:45:55.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:54 smithi033 ceph-mon[25651]: osd.1 [v2:172.21.15.7:6810/1851194794,v1:172.21.15.7:6811/1851194794] boot 2024-04-15T21:45:55.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:54 smithi033 ceph-mon[25651]: osdmap e13: 2 total, 2 up, 2 in 2024-04-15T21:45:55.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:54 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:45:55.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[24826]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:45:55.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[24826]: osdmap e14: 2 total, 2 up, 2 in 2024-04-15T21:45:55.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[24826]: Detected new or changed devices on smithi007 2024-04-15T21:45:55.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:55.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:45:55.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:45:55.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[24826]: Adjusting osd_memory_target on smithi007 to 8034M 2024-04-15T21:45:55.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:55.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:55.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:55.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:55.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:55.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:55.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[29541]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:45:55.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[29541]: osdmap e14: 2 total, 2 up, 2 in 2024-04-15T21:45:55.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[29541]: Detected new or changed devices on smithi007 2024-04-15T21:45:55.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:55.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:45:55.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:45:55.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[29541]: Adjusting osd_memory_target on smithi007 to 8034M 2024-04-15T21:45:55.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:55.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:55.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:55.795 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:55.795 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:55.795 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:55 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:56.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:55 smithi033 ceph-mon[25651]: pgmap v54: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:45:56.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:55 smithi033 ceph-mon[25651]: osdmap e14: 2 total, 2 up, 2 in 2024-04-15T21:45:56.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:55 smithi033 ceph-mon[25651]: Detected new or changed devices on smithi007 2024-04-15T21:45:56.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:55 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:56.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:55 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:45:56.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:55 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:45:56.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:55 smithi033 ceph-mon[25651]: Adjusting osd_memory_target on smithi007 to 8034M 2024-04-15T21:45:56.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:55 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:56.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:55 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:56.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:55 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:45:56.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:55 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:56.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:55 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:45:56.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:55 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:45:57.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:57 smithi007 ceph-mon[24826]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:45:57.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:57 smithi007 ceph-mon[24826]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:45:57.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:45:57.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:45:57.909 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:45:57 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:57.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:57 smithi007 ceph-mon[29541]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:45:57.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:57 smithi007 ceph-mon[29541]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:45:57.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:45:57.909 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:45:57.910 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:45:57 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:45:58.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:57 smithi033 ceph-mon[25651]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:45:58.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:57 smithi033 ceph-mon[25651]: from='client.24178 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:45:58.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:57 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:45:58.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:57 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:45:58.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:45:57 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:00.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:00 smithi033 ceph-mon[25651]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:00.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:00 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1656091214' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "668b55eb-c95a-46d6-8e89-718b5d394d89"}]: dispatch 2024-04-15T21:46:00.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:00 smithi033 ceph-mon[25651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "668b55eb-c95a-46d6-8e89-718b5d394d89"}]: dispatch 2024-04-15T21:46:00.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:00 smithi033 ceph-mon[25651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "668b55eb-c95a-46d6-8e89-718b5d394d89"}]': finished 2024-04-15T21:46:00.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:00 smithi033 ceph-mon[25651]: osdmap e15: 3 total, 2 up, 3 in 2024-04-15T21:46:00.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:00 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:00.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:00 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3615125562' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:46:00.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[24826]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:00.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1656091214' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "668b55eb-c95a-46d6-8e89-718b5d394d89"}]: dispatch 2024-04-15T21:46:00.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[24826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "668b55eb-c95a-46d6-8e89-718b5d394d89"}]: dispatch 2024-04-15T21:46:00.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[24826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "668b55eb-c95a-46d6-8e89-718b5d394d89"}]': finished 2024-04-15T21:46:00.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[24826]: osdmap e15: 3 total, 2 up, 3 in 2024-04-15T21:46:00.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:00.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3615125562' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:46:00.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[29541]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:00.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1656091214' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "668b55eb-c95a-46d6-8e89-718b5d394d89"}]: dispatch 2024-04-15T21:46:00.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[29541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "668b55eb-c95a-46d6-8e89-718b5d394d89"}]: dispatch 2024-04-15T21:46:00.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[29541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "668b55eb-c95a-46d6-8e89-718b5d394d89"}]': finished 2024-04-15T21:46:00.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[29541]: osdmap e15: 3 total, 2 up, 3 in 2024-04-15T21:46:00.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:00.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:00 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3615125562' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:46:02.712 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:02 smithi007 ceph-mon[24826]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:02.713 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:02 smithi007 ceph-mon[29541]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:02.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:02 smithi033 ceph-mon[25651]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:04.713 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:04 smithi007 ceph-mon[24826]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:04.714 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:04 smithi007 ceph-mon[29541]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:04.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:04 smithi033 ceph-mon[25651]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:06.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:06 smithi033 ceph-mon[25651]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:06.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-15T21:46:06.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:06.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:06 smithi007 ceph-mon[24826]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:06.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-15T21:46:06.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:06.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:06 smithi007 ceph-mon[29541]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:06.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-15T21:46:06.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:07.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:07 smithi033 ceph-mon[25651]: Deploying daemon osd.2 on smithi007 2024-04-15T21:46:07.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:07 smithi007 ceph-mon[24826]: Deploying daemon osd.2 on smithi007 2024-04-15T21:46:07.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:07 smithi007 ceph-mon[29541]: Deploying daemon osd.2 on smithi007 2024-04-15T21:46:08.710 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:08 smithi007 ceph-mon[24826]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:08.710 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:08 smithi007 ceph-mon[29541]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:08.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:08 smithi033 ceph-mon[25651]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:10.588 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:10 smithi007 ceph-mon[24826]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:10.589 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:10 smithi007 ceph-mon[29541]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:10.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:10 smithi033 ceph-mon[25651]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:11.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:11 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:11.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:11 smithi007 ceph-mon[24826]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:11.940 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:11 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:11.941 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:11 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:11.941 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:11 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:11.941 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:11 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:11.941 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:11 smithi007 ceph-mon[29541]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:11.941 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:11 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:11.941 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:11 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:11.941 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:11 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:11 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:11 smithi033 ceph-mon[25651]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:11 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:11 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:12.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:11 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:13.015 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:12 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:46:13.016 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:12 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:46:13.016 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:12 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:46:13.016 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:12 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:46:13.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:12 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:46:13.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:12 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:46:13.581 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 2 on host 'smithi007' 2024-04-15T21:46:13.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:13 smithi007 ceph-mon[29541]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:13.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:13.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:13 smithi007 ceph-mon[24826]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:13.871 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:14.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:13 smithi033 ceph-mon[25651]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:14.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:14.101 DEBUG:teuthology.orchestra.run.smithi007:osd.2> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.2.service 2024-04-15T21:46:14.104 INFO:tasks.cephadm:Deploying osd.3 on smithi007 with /dev/vg_nvme/lv_1... 2024-04-15T21:46:14.104 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-15T21:46:14.740 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:14 smithi007 ceph-mon[24826]: from='osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T21:46:14.740 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:14 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:14.740 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:14 smithi007 ceph-mon[29541]: from='osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T21:46:14.740 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:14 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:15.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:14 smithi033 ceph-mon[25651]: from='osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-15T21:46:15.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:14 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:15.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:15 smithi007 ceph-mon[24826]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:15.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:15 smithi007 ceph-mon[24826]: from='osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T21:46:15.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:15 smithi007 ceph-mon[24826]: osdmap e16: 3 total, 2 up, 3 in 2024-04-15T21:46:15.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:15 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:15.903 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:15 smithi007 ceph-mon[24826]: from='osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:46:15.904 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:15 smithi007 ceph-mon[29541]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:15.904 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:15 smithi007 ceph-mon[29541]: from='osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T21:46:15.904 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:15 smithi007 ceph-mon[29541]: osdmap e16: 3 total, 2 up, 3 in 2024-04-15T21:46:15.904 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:15 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:15.904 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:15 smithi007 ceph-mon[29541]: from='osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:46:15.905 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:46:15 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2[38306]: 2024-04-15T21:46:15.676+0000 7f24fe59a700 -1 osd.2 0 waiting for initial osdmap 2024-04-15T21:46:15.905 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:46:15 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2[38306]: 2024-04-15T21:46:15.680+0000 7f24f9f33700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T21:46:15.967 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-15T21:46:15.968 INFO:teuthology.orchestra.run.smithi007.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-15T21:46:15.968 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 10+0 records in 2024-04-15T21:46:15.968 INFO:teuthology.orchestra.run.smithi007.stderr:10+0 records out 2024-04-15T21:46:15.968 INFO:teuthology.orchestra.run.smithi007.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00978887 s, 1.1 GB/s 2024-04-15T21:46:15.968 INFO:teuthology.orchestra.run.smithi007.stderr: stderr: 2024-04-15T21:46:15.969 INFO:teuthology.orchestra.run.smithi007.stderr:--> Zapping successful for: 2024-04-15T21:46:16.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:15 smithi033 ceph-mon[25651]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:16.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:15 smithi033 ceph-mon[25651]: from='osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-15T21:46:16.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:15 smithi033 ceph-mon[25651]: osdmap e16: 3 total, 2 up, 3 in 2024-04-15T21:46:16.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:15 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:16.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:15 smithi033 ceph-mon[25651]: from='osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:46:16.921 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:16 smithi007 ceph-mon[24826]: from='osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-15T21:46:16.921 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:16 smithi007 ceph-mon[24826]: osdmap e17: 3 total, 2 up, 3 in 2024-04-15T21:46:16.921 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:16 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:16.921 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:16 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:16.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:16 smithi007 ceph-mon[29541]: from='osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-15T21:46:16.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:16 smithi007 ceph-mon[29541]: osdmap e17: 3 total, 2 up, 3 in 2024-04-15T21:46:16.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:16 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:16.922 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:16 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:17.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:16 smithi033 ceph-mon[25651]: from='osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-15T21:46:17.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:16 smithi033 ceph-mon[25651]: osdmap e17: 3 total, 2 up, 3 in 2024-04-15T21:46:17.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:16 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:17.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:16 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:17.526 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi007:vg_nvme/lv_1 2024-04-15T21:46:17.790 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[24826]: purged_snaps scrub starts 2024-04-15T21:46:17.790 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[24826]: purged_snaps scrub ok 2024-04-15T21:46:17.790 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[24826]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:17.790 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:17.790 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[24826]: osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948] boot 2024-04-15T21:46:17.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[24826]: osdmap e18: 3 total, 3 up, 3 in 2024-04-15T21:46:17.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:17.791 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[29541]: purged_snaps scrub starts 2024-04-15T21:46:17.791 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[29541]: purged_snaps scrub ok 2024-04-15T21:46:17.791 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[29541]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:17.791 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:17.791 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[29541]: osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948] boot 2024-04-15T21:46:17.791 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[29541]: osdmap e18: 3 total, 3 up, 3 in 2024-04-15T21:46:17.791 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:17 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:18.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:17 smithi033 ceph-mon[25651]: purged_snaps scrub starts 2024-04-15T21:46:18.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:17 smithi033 ceph-mon[25651]: purged_snaps scrub ok 2024-04-15T21:46:18.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:17 smithi033 ceph-mon[25651]: pgmap v69: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-15T21:46:18.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:17 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:18.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:17 smithi033 ceph-mon[25651]: osd.2 [v2:172.21.15.7:6818/3569809948,v1:172.21.15.7:6819/3569809948] boot 2024-04-15T21:46:18.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:17 smithi033 ceph-mon[25651]: osdmap e18: 3 total, 3 up, 3 in 2024-04-15T21:46:18.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:17 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:46:19.244 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[24826]: osdmap e19: 3 total, 3 up, 3 in 2024-04-15T21:46:19.244 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[24826]: Detected new or changed devices on smithi007 2024-04-15T21:46:19.244 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:19.244 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:19.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:19.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:19.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[24826]: Adjusting osd_memory_target on smithi007 to 5356M 2024-04-15T21:46:19.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:19.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:19.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:19.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:19.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:19.245 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:19.245 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[29541]: osdmap e19: 3 total, 3 up, 3 in 2024-04-15T21:46:19.245 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[29541]: Detected new or changed devices on smithi007 2024-04-15T21:46:19.245 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:19.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:19.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:19.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:19.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[29541]: Adjusting osd_memory_target on smithi007 to 5356M 2024-04-15T21:46:19.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:19.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:19.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:19.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:19.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:19.246 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:19.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:18 smithi033 ceph-mon[25651]: osdmap e19: 3 total, 3 up, 3 in 2024-04-15T21:46:19.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:18 smithi033 ceph-mon[25651]: Detected new or changed devices on smithi007 2024-04-15T21:46:19.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:19.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:19.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:19.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:19.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:18 smithi033 ceph-mon[25651]: Adjusting osd_memory_target on smithi007 to 5356M 2024-04-15T21:46:19.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:19.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:19.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:19.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:19.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:19.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:20.257 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[29541]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:20.257 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-15T21:46:20.258 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-15T21:46:20.258 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[29541]: osdmap e20: 3 total, 3 up, 3 in 2024-04-15T21:46:20.258 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T21:46:20.258 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:46:20.258 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:46:20.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[24826]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:20.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-15T21:46:20.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-15T21:46:20.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[24826]: osdmap e20: 3 total, 3 up, 3 in 2024-04-15T21:46:20.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T21:46:20.258 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:46:20.259 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:19 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:46:20.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:19 smithi033 ceph-mon[25651]: pgmap v72: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:20.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:19 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-15T21:46:20.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:19 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-15T21:46:20.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:19 smithi033 ceph-mon[25651]: osdmap e20: 3 total, 3 up, 3 in 2024-04-15T21:46:20.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:19 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-15T21:46:20.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:19 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:46:20.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:19 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:46:21.250 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:46:21 smithi007 sudo[39744]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T21:46:21.250 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:46:21 smithi007 sudo[39744]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T21:46:21.250 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:46:21 smithi007 sudo[39744]: pam_unix(sudo:session): session closed for user root 2024-04-15T21:46:21.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:20 smithi007 ceph-mon[24826]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:46:21.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:20 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:21.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:20 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-15T21:46:21.251 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:20 smithi007 ceph-mon[24826]: osdmap e21: 3 total, 3 up, 3 in 2024-04-15T21:46:21.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:20 smithi007 ceph-mon[29541]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:46:21.251 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:20 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:21.252 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:20 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-15T21:46:21.252 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:20 smithi007 ceph-mon[29541]: osdmap e21: 3 total, 3 up, 3 in 2024-04-15T21:46:21.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:20 smithi033 ceph-mon[25651]: from='client.14301 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi007:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:46:21.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:20 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:21.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:20 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-15T21:46:21.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:20 smithi033 ceph-mon[25651]: osdmap e21: 3 total, 3 up, 3 in 2024-04-15T21:46:21.504 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:46:21 smithi007 sudo[39768]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T21:46:21.505 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:46:21 smithi007 sudo[39768]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T21:46:21.505 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:46:21 smithi007 sudo[39768]: pam_unix(sudo:session): session closed for user root 2024-04-15T21:46:21.792 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:46:21 smithi007 sudo[39777]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T21:46:21.792 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:46:21 smithi007 sudo[39777]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T21:46:21.792 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:46:21 smithi007 sudo[39777]: pam_unix(sudo:session): session closed for user root 2024-04-15T21:46:22.163 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:46:21 smithi007 sudo[39793]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T21:46:22.164 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:46:21 smithi007 sudo[39793]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T21:46:22.164 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:46:21 smithi007 sudo[39793]: pam_unix(sudo:session): session closed for user root 2024-04-15T21:46:22.432 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:46:22 smithi007 sudo[39819]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T21:46:22.432 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:46:22 smithi007 sudo[39819]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T21:46:22.433 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:46:22 smithi007 sudo[39819]: pam_unix(sudo:session): session closed for user root 2024-04-15T21:46:22.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[24826]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:22.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:22.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[24826]: osdmap e22: 3 total, 3 up, 3 in 2024-04-15T21:46:22.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3302498142' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26df7960-b04a-41c8-8f00-29c0bd10806f"}]: dispatch 2024-04-15T21:46:22.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3302498142' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26df7960-b04a-41c8-8f00-29c0bd10806f"}]': finished 2024-04-15T21:46:22.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[24826]: osdmap e23: 4 total, 3 up, 4 in 2024-04-15T21:46:22.433 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:22.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[29541]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:22.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:22.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[29541]: osdmap e22: 3 total, 3 up, 3 in 2024-04-15T21:46:22.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3302498142' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26df7960-b04a-41c8-8f00-29c0bd10806f"}]: dispatch 2024-04-15T21:46:22.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3302498142' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26df7960-b04a-41c8-8f00-29c0bd10806f"}]': finished 2024-04-15T21:46:22.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[29541]: osdmap e23: 4 total, 3 up, 4 in 2024-04-15T21:46:22.434 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:22 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:22.697 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:46:22 smithi007 sudo[39834]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T21:46:22.697 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:46:22 smithi007 sudo[39834]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T21:46:22.697 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:46:22 smithi007 sudo[39834]: pam_unix(sudo:session): session closed for user root 2024-04-15T21:46:22.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:22 smithi033 ceph-mon[25651]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:22.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:22 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:22.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:22 smithi033 ceph-mon[25651]: osdmap e22: 3 total, 3 up, 3 in 2024-04-15T21:46:22.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:22 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3302498142' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "26df7960-b04a-41c8-8f00-29c0bd10806f"}]: dispatch 2024-04-15T21:46:22.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:22 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3302498142' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26df7960-b04a-41c8-8f00-29c0bd10806f"}]': finished 2024-04-15T21:46:22.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:22 smithi033 ceph-mon[25651]: osdmap e23: 4 total, 3 up, 4 in 2024-04-15T21:46:22.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:22 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:22.964 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:22 smithi007 sudo[39857]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T21:46:22.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:22 smithi007 sudo[39857]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T21:46:22.965 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:22 smithi007 sudo[39857]: pam_unix(sudo:session): session closed for user root 2024-04-15T21:46:23.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:22 smithi007 sudo[39908]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T21:46:23.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:22 smithi007 sudo[39908]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T21:46:23.230 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:22 smithi007 sudo[39908]: pam_unix(sudo:session): session closed for user root 2024-04-15T21:46:23.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:23 smithi007 sudo[39912]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T21:46:23.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:23 smithi007 sudo[39912]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T21:46:23.504 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:23 smithi007 sudo[39912]: pam_unix(sudo:session): session closed for user root 2024-04-15T21:46:23.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:23 smithi007 sudo[40102]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T21:46:23.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:23 smithi007 sudo[40102]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T21:46:23.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:23 smithi007 sudo[40102]: pam_unix(sudo:session): session closed for user root 2024-04-15T21:46:24.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:23 smithi033 sudo[27170]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-15T21:46:24.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:23 smithi033 sudo[27170]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T21:46:24.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:23 smithi033 sudo[27170]: pam_unix(sudo:session): session closed for user root 2024-04-15T21:46:24.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[24826]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T21:46:24.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/485059699' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:46:24.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[24826]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T21:46:24.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:46:24.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:46:24.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:46:24.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:46:24.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:46:24.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:46:24.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[29541]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T21:46:24.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/485059699' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:46:24.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[29541]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T21:46:24.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:46:24.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:46:24.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:46:24.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:46:24.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:46:24.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:24 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:46:24.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:24 smithi033 sudo[27173]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-15T21:46:24.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:24 smithi033 sudo[27173]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-15T21:46:24.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:24 smithi033 sudo[27173]: pam_unix(sudo:session): session closed for user root 2024-04-15T21:46:24.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:24 smithi033 ceph-mon[25651]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T21:46:24.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:24 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/485059699' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:46:24.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:24 smithi033 ceph-mon[25651]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T21:46:24.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:24 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:46:24.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:24 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:46:24.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:24 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:46:24.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:24 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:46:24.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:24 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:46:24.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:24 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:46:25.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[24826]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:25.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[24826]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T21:46:25.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[24826]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T21:46:25.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[24826]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T21:46:25.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:46:25.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:46:25.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:46:25.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[24826]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T21:46:25.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[24826]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-15T21:46:25.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:25.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[29541]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:25.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[29541]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T21:46:25.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[29541]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T21:46:25.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[29541]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T21:46:25.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:46:25.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:46:25.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:46:25.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[29541]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T21:46:25.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[29541]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-15T21:46:25.295 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:25 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:25.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:25 smithi033 ceph-mon[25651]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:25.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:25 smithi033 ceph-mon[25651]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T21:46:25.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:25 smithi033 ceph-mon[25651]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T21:46:25.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:25 smithi033 ceph-mon[25651]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-15T21:46:25.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:25 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:46:25.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:25 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:46:25.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:25 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:46:25.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:25 smithi033 ceph-mon[25651]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-15T21:46:25.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:25 smithi033 ceph-mon[25651]: mgrmap e15: y(active, since 2m), standbys: x 2024-04-15T21:46:25.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:25 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:26.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:26 smithi007 ceph-mon[24826]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:26.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:26 smithi007 ceph-mon[29541]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:26.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:26 smithi033 ceph-mon[25651]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:28.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:28 smithi007 ceph-mon[24826]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:28.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:28 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-15T21:46:28.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:28 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:28.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:28 smithi007 ceph-mon[29541]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:28.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:28 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-15T21:46:28.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:28 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:28.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:28 smithi033 ceph-mon[25651]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:28.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:28 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-15T21:46:28.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:28 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:29.675 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:29 smithi007 ceph-mon[24826]: Deploying daemon osd.3 on smithi007 2024-04-15T21:46:29.676 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:29 smithi007 ceph-mon[29541]: Deploying daemon osd.3 on smithi007 2024-04-15T21:46:29.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:29 smithi033 ceph-mon[25651]: Deploying daemon osd.3 on smithi007 2024-04-15T21:46:30.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:30 smithi007 ceph-mon[29541]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:30.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:30 smithi007 ceph-mon[24826]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:30 smithi033 ceph-mon[25651]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:32 smithi033 ceph-mon[25651]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-15T21:46:32.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:32 smithi007 ceph-mon[24826]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-15T21:46:32.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:32 smithi007 ceph-mon[29541]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-15T21:46:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:34 smithi033 ceph-mon[25651]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-15T21:46:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:34 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:34 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:34 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:34 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:34.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:34 smithi007 ceph-mon[24826]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-15T21:46:34.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:34 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:34.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:34 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:34.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:34 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:34.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:34 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:34.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:34 smithi007 ceph-mon[29541]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-15T21:46:34.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:34 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:34.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:34 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:34.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:34 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:34.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:34 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:36.426 INFO:teuthology.orchestra.run.smithi007.stdout:Created osd(s) 3 on host 'smithi007' 2024-04-15T21:46:36.639 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:36 smithi007 ceph-mon[29541]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-15T21:46:36.640 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:36 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:36.640 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:36 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:36.640 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:36 smithi007 ceph-mon[29541]: from='osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T21:46:36.640 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:36 smithi007 ceph-mon[24826]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-15T21:46:36.640 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:36 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:36.640 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:36 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:36.640 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:36 smithi007 ceph-mon[24826]: from='osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T21:46:36.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:36 smithi033 ceph-mon[25651]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-15T21:46:36.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:36 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:36.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:36 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:36.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:36 smithi033 ceph-mon[25651]: from='osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-15T21:46:37.029 DEBUG:teuthology.orchestra.run.smithi007:osd.3> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.3.service 2024-04-15T21:46:37.031 INFO:tasks.cephadm:Deploying osd.4 on smithi033 with /dev/vg_nvme/lv_4... 2024-04-15T21:46:37.031 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-15T21:46:37.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:37.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:37.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:37.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:37.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[29541]: from='osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T21:46:37.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[29541]: osdmap e24: 4 total, 3 up, 4 in 2024-04-15T21:46:37.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:37.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[29541]: from='osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:46:37.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:37.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:37.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:37.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:37.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[24826]: from='osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T21:46:37.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[24826]: osdmap e24: 4 total, 3 up, 4 in 2024-04-15T21:46:37.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:37.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:37 smithi007 ceph-mon[24826]: from='osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:46:37.651 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:37 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:37.652 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:37 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:37.652 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:37 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:37.652 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:37 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:37.652 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:37 smithi033 ceph-mon[25651]: from='osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-15T21:46:37.652 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:37 smithi033 ceph-mon[25651]: osdmap e24: 4 total, 3 up, 4 in 2024-04-15T21:46:37.652 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:37 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:37.652 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:37 smithi033 ceph-mon[25651]: from='osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]: dispatch 2024-04-15T21:46:38.248 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:46:38 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3[41290]: 2024-04-15T21:46:38.060+0000 7fd8b519d700 -1 osd.3 0 waiting for initial osdmap 2024-04-15T21:46:38.248 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:46:38 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3[41290]: 2024-04-15T21:46:38.063+0000 7fd8afb34700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T21:46:38.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:38 smithi007 ceph-mon[24826]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-15T21:46:38.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:38 smithi007 ceph-mon[24826]: from='osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-15T21:46:38.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:38 smithi007 ceph-mon[24826]: osdmap e25: 4 total, 3 up, 4 in 2024-04-15T21:46:38.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:38 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:38.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:38 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:38.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:38 smithi007 ceph-mon[29541]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-15T21:46:38.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:38 smithi007 ceph-mon[29541]: from='osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-15T21:46:38.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:38 smithi007 ceph-mon[29541]: osdmap e25: 4 total, 3 up, 4 in 2024-04-15T21:46:38.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:38 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:38.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:38 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:38.578 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:38 smithi033 ceph-mon[25651]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-15T21:46:38.578 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:38 smithi033 ceph-mon[25651]: from='osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi007", "root=default"]}]': finished 2024-04-15T21:46:38.578 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:38 smithi033 ceph-mon[25651]: osdmap e25: 4 total, 3 up, 4 in 2024-04-15T21:46:38.578 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:38 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:38.579 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:38 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:39.078 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-15T21:46:39.079 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-15T21:46:39.079 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-04-15T21:46:39.079 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-04-15T21:46:39.079 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111712 s, 939 MB/s 2024-04-15T21:46:39.079 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-04-15T21:46:39.834 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_4 2024-04-15T21:46:40.325 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:40 smithi033 ceph-mon[25651]: purged_snaps scrub starts 2024-04-15T21:46:40.325 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:40 smithi033 ceph-mon[25651]: purged_snaps scrub ok 2024-04-15T21:46:40.325 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:40 smithi033 ceph-mon[25651]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:40.325 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:40 smithi033 ceph-mon[25651]: osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749] boot 2024-04-15T21:46:40.325 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:40 smithi033 ceph-mon[25651]: osdmap e26: 4 total, 4 up, 4 in 2024-04-15T21:46:40.325 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:40 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:40.325 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:40 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:40.325 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:40 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:40.325 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:40 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:40.325 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:40 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:40.325 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:40 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:40.325 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:40 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:40.325 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:40 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:40.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[24826]: purged_snaps scrub starts 2024-04-15T21:46:40.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[24826]: purged_snaps scrub ok 2024-04-15T21:46:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[24826]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[24826]: osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749] boot 2024-04-15T21:46:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[24826]: osdmap e26: 4 total, 4 up, 4 in 2024-04-15T21:46:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:40.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:40.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[29541]: purged_snaps scrub starts 2024-04-15T21:46:40.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[29541]: purged_snaps scrub ok 2024-04-15T21:46:40.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[29541]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-04-15T21:46:40.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[29541]: osd.3 [v2:172.21.15.7:6826/2388023749,v1:172.21.15.7:6827/2388023749] boot 2024-04-15T21:46:40.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[29541]: osdmap e26: 4 total, 4 up, 4 in 2024-04-15T21:46:40.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:46:40.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:40.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:40.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:40.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:40.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:46:40.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:40.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:40 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:41.322 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:41 smithi033 ceph-mon[25651]: Detected new or changed devices on smithi007 2024-04-15T21:46:41.322 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:41 smithi033 ceph-mon[25651]: Adjusting osd_memory_target on smithi007 to 4017M 2024-04-15T21:46:41.323 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:41 smithi033 ceph-mon[25651]: osdmap e27: 4 total, 4 up, 4 in 2024-04-15T21:46:41.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:41 smithi007 ceph-mon[29541]: Detected new or changed devices on smithi007 2024-04-15T21:46:41.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:41 smithi007 ceph-mon[29541]: Adjusting osd_memory_target on smithi007 to 4017M 2024-04-15T21:46:41.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:41 smithi007 ceph-mon[29541]: osdmap e27: 4 total, 4 up, 4 in 2024-04-15T21:46:41.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:41 smithi007 ceph-mon[24826]: Detected new or changed devices on smithi007 2024-04-15T21:46:41.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:41 smithi007 ceph-mon[24826]: Adjusting osd_memory_target on smithi007 to 4017M 2024-04-15T21:46:41.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:41 smithi007 ceph-mon[24826]: osdmap e27: 4 total, 4 up, 4 in 2024-04-15T21:46:42.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:42 smithi033 ceph-mon[25651]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:42.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:42 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:46:42.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:42 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:46:42.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:42 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:42.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:42 smithi007 ceph-mon[24826]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:42.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:42 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:46:42.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:42 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:46:42.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:42 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:42.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:42 smithi007 ceph-mon[29541]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:42.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:42 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:46:42.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:42 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:46:42.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:42 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:43.335 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:43 smithi033 ceph-mon[25651]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:46:43.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:43 smithi007 ceph-mon[24826]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:46:43.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:43 smithi007 ceph-mon[29541]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:46:44.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:44 smithi033 ceph-mon[25651]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:44.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:44 smithi033 ceph-mon[25651]: from='client.? 172.21.15.33:0/3330015733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c02631-9af8-4210-81dc-78b3e09d8248"}]: dispatch 2024-04-15T21:46:44.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:44 smithi033 ceph-mon[25651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c02631-9af8-4210-81dc-78b3e09d8248"}]: dispatch 2024-04-15T21:46:44.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:44 smithi033 ceph-mon[25651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3c02631-9af8-4210-81dc-78b3e09d8248"}]': finished 2024-04-15T21:46:44.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:44 smithi033 ceph-mon[25651]: osdmap e28: 5 total, 4 up, 5 in 2024-04-15T21:46:44.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:44 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:46:44.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:44 smithi007 ceph-mon[24826]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:44.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:44 smithi007 ceph-mon[24826]: from='client.? 172.21.15.33:0/3330015733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c02631-9af8-4210-81dc-78b3e09d8248"}]: dispatch 2024-04-15T21:46:44.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:44 smithi007 ceph-mon[24826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c02631-9af8-4210-81dc-78b3e09d8248"}]: dispatch 2024-04-15T21:46:44.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:44 smithi007 ceph-mon[24826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3c02631-9af8-4210-81dc-78b3e09d8248"}]': finished 2024-04-15T21:46:44.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:44 smithi007 ceph-mon[24826]: osdmap e28: 5 total, 4 up, 5 in 2024-04-15T21:46:44.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:44 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:46:44.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:44 smithi007 ceph-mon[29541]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:44.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:44 smithi007 ceph-mon[29541]: from='client.? 172.21.15.33:0/3330015733' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c02631-9af8-4210-81dc-78b3e09d8248"}]: dispatch 2024-04-15T21:46:44.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:44 smithi007 ceph-mon[29541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a3c02631-9af8-4210-81dc-78b3e09d8248"}]: dispatch 2024-04-15T21:46:44.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:44 smithi007 ceph-mon[29541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a3c02631-9af8-4210-81dc-78b3e09d8248"}]': finished 2024-04-15T21:46:44.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:44 smithi007 ceph-mon[29541]: osdmap e28: 5 total, 4 up, 5 in 2024-04-15T21:46:44.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:44 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:46:45.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:45 smithi033 ceph-mon[25651]: from='client.? 172.21.15.33:0/1221057428' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:46:45.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:45 smithi007 ceph-mon[24826]: from='client.? 172.21.15.33:0/1221057428' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:46:45.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:45 smithi007 ceph-mon[29541]: from='client.? 172.21.15.33:0/1221057428' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:46:46.711 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:46 smithi007 ceph-mon[24826]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:46.711 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:46 smithi007 ceph-mon[29541]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:46.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:46 smithi033 ceph-mon[25651]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:48.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:48 smithi033 ceph-mon[25651]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:48.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:48 smithi007 ceph-mon[24826]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:48.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:48 smithi007 ceph-mon[29541]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:50.685 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:50 smithi033 ceph-mon[25651]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:50.686 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:50 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-15T21:46:50.686 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:50 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:50.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:50 smithi007 ceph-mon[24826]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:50.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:50 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-15T21:46:50.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:50 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:50.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:50 smithi007 ceph-mon[29541]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:50.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:50 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-15T21:46:50.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:50 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:51.699 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:51 smithi033 ceph-mon[25651]: Deploying daemon osd.4 on smithi033 2024-04-15T21:46:51.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:51 smithi007 ceph-mon[24826]: Deploying daemon osd.4 on smithi033 2024-04-15T21:46:51.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:51 smithi007 ceph-mon[29541]: Deploying daemon osd.4 on smithi033 2024-04-15T21:46:52.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:52 smithi033 ceph-mon[25651]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:52.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:52 smithi007 ceph-mon[24826]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:52.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:52 smithi007 ceph-mon[29541]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:54.742 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:54 smithi033 ceph-mon[25651]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:54.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:54 smithi007 ceph-mon[24826]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:54.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:54 smithi007 ceph-mon[29541]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:56.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:55 smithi033 ceph-mon[25651]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:56.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:55 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:56.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:55 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:56.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:55 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:56.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:55 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:56.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:55 smithi007 ceph-mon[24826]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:56.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:55 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:56.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:55 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:56.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:55 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:56.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:55 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:56.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:55 smithi007 ceph-mon[29541]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:56.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:55 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:56.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:55 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:56.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:55 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:56.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:55 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:58.182 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 4 on host 'smithi033' 2024-04-15T21:46:58.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:58 smithi033 ceph-mon[25651]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:58.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:58 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:58.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:58 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:58.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:58 smithi033 ceph-mon[25651]: from='osd.4 [v2:172.21.15.33:6800/993063057,v1:172.21.15.33:6801/993063057]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T21:46:58.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:58 smithi033 ceph-mon[25651]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T21:46:58.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:58 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:58.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:58 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:58.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:58 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:58.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:58 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:58.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[24826]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:58.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:58.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:58.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[24826]: from='osd.4 [v2:172.21.15.33:6800/993063057,v1:172.21.15.33:6801/993063057]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T21:46:58.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[24826]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T21:46:58.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:58.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:58.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:58.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:58.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[29541]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:46:58.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:58.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:58.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[29541]: from='osd.4 [v2:172.21.15.33:6800/993063057,v1:172.21.15.33:6801/993063057]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T21:46:58.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[29541]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-15T21:46:58.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:46:58.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:46:58.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:46:58.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:58 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:46:58.802 DEBUG:teuthology.orchestra.run.smithi033:osd.4> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.4.service 2024-04-15T21:46:58.805 INFO:tasks.cephadm:Deploying osd.5 on smithi033 with /dev/vg_nvme/lv_3... 2024-04-15T21:46:58.805 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-15T21:46:59.903 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:59 smithi033 ceph-mon[25651]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-15T21:46:59.903 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:59 smithi033 ceph-mon[25651]: osdmap e29: 5 total, 4 up, 5 in 2024-04-15T21:46:59.903 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:59 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:46:59.903 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:59 smithi033 ceph-mon[25651]: from='osd.4 [v2:172.21.15.33:6800/993063057,v1:172.21.15.33:6801/993063057]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:46:59.904 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:46:59 smithi033 ceph-mon[25651]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:46:59.904 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:46:59 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:46:59.575+0000 7fcbfb5d8700 -1 osd.4 0 waiting for initial osdmap 2024-04-15T21:46:59.904 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:46:59 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:46:59.579+0000 7fcbf476c700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T21:47:00.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:59 smithi007 ceph-mon[24826]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-15T21:47:00.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:59 smithi007 ceph-mon[24826]: osdmap e29: 5 total, 4 up, 5 in 2024-04-15T21:47:00.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:59 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:47:00.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:59 smithi007 ceph-mon[24826]: from='osd.4 [v2:172.21.15.33:6800/993063057,v1:172.21.15.33:6801/993063057]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:00.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:46:59 smithi007 ceph-mon[24826]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:00.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:59 smithi007 ceph-mon[29541]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-15T21:47:00.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:59 smithi007 ceph-mon[29541]: osdmap e29: 5 total, 4 up, 5 in 2024-04-15T21:47:00.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:59 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:47:00.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:59 smithi007 ceph-mon[29541]: from='osd.4 [v2:172.21.15.33:6800/993063057,v1:172.21.15.33:6801/993063057]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:00.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:46:59 smithi007 ceph-mon[29541]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:00.841 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:00 smithi033 ceph-mon[25651]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:47:00.841 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:00 smithi033 ceph-mon[25651]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-15T21:47:00.841 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:00 smithi033 ceph-mon[25651]: osdmap e30: 5 total, 4 up, 5 in 2024-04-15T21:47:00.841 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:00 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:47:00.841 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:00 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:47:01.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:00 smithi007 ceph-mon[24826]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:47:01.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:00 smithi007 ceph-mon[24826]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-15T21:47:01.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:00 smithi007 ceph-mon[24826]: osdmap e30: 5 total, 4 up, 5 in 2024-04-15T21:47:01.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:00 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:47:01.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:00 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:47:01.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:00 smithi007 ceph-mon[29541]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-15T21:47:01.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:00 smithi007 ceph-mon[29541]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-15T21:47:01.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:00 smithi007 ceph-mon[29541]: osdmap e30: 5 total, 4 up, 5 in 2024-04-15T21:47:01.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:00 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:47:01.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:00 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:47:01.722 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:01 smithi033 ceph-mon[25651]: purged_snaps scrub starts 2024-04-15T21:47:01.722 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:01 smithi033 ceph-mon[25651]: purged_snaps scrub ok 2024-04-15T21:47:01.722 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:01 smithi033 ceph-mon[25651]: osd.4 [v2:172.21.15.33:6800/993063057,v1:172.21.15.33:6801/993063057] boot 2024-04-15T21:47:01.723 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:01 smithi033 ceph-mon[25651]: osdmap e31: 5 total, 5 up, 5 in 2024-04-15T21:47:01.723 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:01 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:47:01.723 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:01 smithi033 ceph-mon[25651]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:01.723 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:01 smithi033 ceph-mon[25651]: osdmap e32: 5 total, 5 up, 5 in 2024-04-15T21:47:01.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[24826]: purged_snaps scrub starts 2024-04-15T21:47:01.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[24826]: purged_snaps scrub ok 2024-04-15T21:47:01.869 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[24826]: osd.4 [v2:172.21.15.33:6800/993063057,v1:172.21.15.33:6801/993063057] boot 2024-04-15T21:47:01.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[24826]: osdmap e31: 5 total, 5 up, 5 in 2024-04-15T21:47:01.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:47:01.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[24826]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:01.870 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[24826]: osdmap e32: 5 total, 5 up, 5 in 2024-04-15T21:47:01.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[29541]: purged_snaps scrub starts 2024-04-15T21:47:01.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[29541]: purged_snaps scrub ok 2024-04-15T21:47:01.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[29541]: osd.4 [v2:172.21.15.33:6800/993063057,v1:172.21.15.33:6801/993063057] boot 2024-04-15T21:47:01.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[29541]: osdmap e31: 5 total, 5 up, 5 in 2024-04-15T21:47:01.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:47:01.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[29541]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:01.870 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:01 smithi007 ceph-mon[29541]: osdmap e32: 5 total, 5 up, 5 in 2024-04-15T21:47:02.032 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-15T21:47:02.032 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-15T21:47:02.032 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-04-15T21:47:02.032 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-04-15T21:47:02.032 INFO:teuthology.orchestra.run.smithi033.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0121315 s, 864 MB/s 2024-04-15T21:47:02.032 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-04-15T21:47:03.062 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_3 2024-04-15T21:47:03.566 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:03 smithi033 ceph-mon[25651]: Detected new or changed devices on smithi033 2024-04-15T21:47:03.566 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:03 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:03.566 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:03 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:03.566 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:03 smithi033 ceph-mon[25651]: Adjusting osd_memory_target on smithi033 to 17093M 2024-04-15T21:47:03.566 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:03 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:03.566 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:03 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:03.566 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:03 smithi033 ceph-mon[25651]: osdmap e33: 5 total, 5 up, 5 in 2024-04-15T21:47:03.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[24826]: Detected new or changed devices on smithi033 2024-04-15T21:47:03.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:03.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:03.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[24826]: Adjusting osd_memory_target on smithi033 to 17093M 2024-04-15T21:47:03.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:03.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:03.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[24826]: osdmap e33: 5 total, 5 up, 5 in 2024-04-15T21:47:03.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[29541]: Detected new or changed devices on smithi033 2024-04-15T21:47:03.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:03.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:03.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[29541]: Adjusting osd_memory_target on smithi033 to 17093M 2024-04-15T21:47:03.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:03.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:03.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:03 smithi007 ceph-mon[29541]: osdmap e33: 5 total, 5 up, 5 in 2024-04-15T21:47:04.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:04 smithi033 ceph-mon[25651]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:04.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:04 smithi033 ceph-mon[25651]: osdmap e34: 5 total, 5 up, 5 in 2024-04-15T21:47:04.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:04 smithi007 ceph-mon[24826]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:04.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:04 smithi007 ceph-mon[24826]: osdmap e34: 5 total, 5 up, 5 in 2024-04-15T21:47:04.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:04 smithi007 ceph-mon[29541]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:04.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:04 smithi007 ceph-mon[29541]: osdmap e34: 5 total, 5 up, 5 in 2024-04-15T21:47:06.001 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:05 smithi033 ceph-mon[25651]: osdmap e35: 5 total, 5 up, 5 in 2024-04-15T21:47:06.001 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:05 smithi033 ceph-mon[25651]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T21:47:06.001 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:05 smithi033 ceph-mon[25651]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:47:06.001 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:05 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:47:06.001 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:05 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:47:06.001 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:05 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:06.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:05 smithi007 ceph-mon[24826]: osdmap e35: 5 total, 5 up, 5 in 2024-04-15T21:47:06.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:05 smithi007 ceph-mon[24826]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T21:47:06.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:05 smithi007 ceph-mon[24826]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:47:06.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:05 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:47:06.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:05 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:47:06.043 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:05 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:06.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:05 smithi007 ceph-mon[29541]: osdmap e35: 5 total, 5 up, 5 in 2024-04-15T21:47:06.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:05 smithi007 ceph-mon[29541]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-15T21:47:06.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:05 smithi007 ceph-mon[29541]: from='client.24251 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:47:06.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:05 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:47:06.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:05 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:47:06.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:05 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:08.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:07 smithi033 ceph-mon[25651]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-15T21:47:08.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:07 smithi033 ceph-mon[25651]: from='client.? 172.21.15.33:0/3497037478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad528133-1de4-4e6d-af65-fd06b811054b"}]: dispatch 2024-04-15T21:47:08.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:07 smithi033 ceph-mon[25651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad528133-1de4-4e6d-af65-fd06b811054b"}]: dispatch 2024-04-15T21:47:08.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:07 smithi033 ceph-mon[25651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad528133-1de4-4e6d-af65-fd06b811054b"}]': finished 2024-04-15T21:47:08.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:07 smithi033 ceph-mon[25651]: osdmap e36: 6 total, 5 up, 6 in 2024-04-15T21:47:08.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:07 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:08.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:07 smithi007 ceph-mon[24826]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-15T21:47:08.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:07 smithi007 ceph-mon[24826]: from='client.? 172.21.15.33:0/3497037478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad528133-1de4-4e6d-af65-fd06b811054b"}]: dispatch 2024-04-15T21:47:08.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:07 smithi007 ceph-mon[24826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad528133-1de4-4e6d-af65-fd06b811054b"}]: dispatch 2024-04-15T21:47:08.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:07 smithi007 ceph-mon[24826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad528133-1de4-4e6d-af65-fd06b811054b"}]': finished 2024-04-15T21:47:08.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:07 smithi007 ceph-mon[24826]: osdmap e36: 6 total, 5 up, 6 in 2024-04-15T21:47:08.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:07 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:08.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:07 smithi007 ceph-mon[29541]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-04-15T21:47:08.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:07 smithi007 ceph-mon[29541]: from='client.? 172.21.15.33:0/3497037478' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad528133-1de4-4e6d-af65-fd06b811054b"}]: dispatch 2024-04-15T21:47:08.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:07 smithi007 ceph-mon[29541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ad528133-1de4-4e6d-af65-fd06b811054b"}]: dispatch 2024-04-15T21:47:08.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:07 smithi007 ceph-mon[29541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ad528133-1de4-4e6d-af65-fd06b811054b"}]': finished 2024-04-15T21:47:08.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:07 smithi007 ceph-mon[29541]: osdmap e36: 6 total, 5 up, 6 in 2024-04-15T21:47:08.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:07 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:09.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:08 smithi033 ceph-mon[25651]: from='client.? 172.21.15.33:0/1674750147' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:47:09.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:08 smithi007 ceph-mon[24826]: from='client.? 172.21.15.33:0/1674750147' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:47:09.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:08 smithi007 ceph-mon[29541]: from='client.? 172.21.15.33:0/1674750147' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:47:10.668 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:10 smithi033 ceph-mon[25651]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T21:47:10.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:10 smithi007 ceph-mon[24826]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T21:47:10.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:10 smithi007 ceph-mon[29541]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T21:47:12.436 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:12 smithi033 ceph-mon[25651]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-15T21:47:12.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:12 smithi007 ceph-mon[24826]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-15T21:47:12.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:12 smithi007 ceph-mon[29541]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-04-15T21:47:13.715 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:47:13.715 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:47:13.715 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-15T21:47:13.716 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:13.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:47:13.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:47:13.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-15T21:47:13.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:13.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:47:13.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:47:13.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-15T21:47:13.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:14.729 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:14 smithi033 ceph-mon[25651]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:14.729 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:14 smithi033 ceph-mon[25651]: Deploying daemon osd.5 on smithi033 2024-04-15T21:47:14.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:14 smithi007 ceph-mon[24826]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:14.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:14 smithi007 ceph-mon[24826]: Deploying daemon osd.5 on smithi033 2024-04-15T21:47:14.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:14 smithi007 ceph-mon[29541]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:14.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:14 smithi007 ceph-mon[29541]: Deploying daemon osd.5 on smithi033 2024-04-15T21:47:16.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:16 smithi033 ceph-mon[25651]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:16.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:16 smithi007 ceph-mon[24826]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:16.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:16 smithi007 ceph-mon[29541]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:18.689 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:18 smithi033 ceph-mon[25651]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:18.690 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:18.690 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:47:18.690 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:18.690 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:18 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:47:18.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:18 smithi007 ceph-mon[24826]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:18.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:18.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:47:18.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:18.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:18 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:47:18.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:18 smithi007 ceph-mon[29541]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:18.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:18.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:47:18.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:18.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:18 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:47:20.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:20 smithi033 ceph-mon[25651]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:20.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:20 smithi007 ceph-mon[24826]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:20.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:20 smithi007 ceph-mon[29541]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:20.931 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 5 on host 'smithi033' 2024-04-15T21:47:21.534 DEBUG:teuthology.orchestra.run.smithi033:osd.5> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.5.service 2024-04-15T21:47:21.536 INFO:tasks.cephadm:Deploying osd.6 on smithi033 with /dev/vg_nvme/lv_2... 2024-04-15T21:47:21.537 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-15T21:47:22.209 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:21 smithi033 ceph-mon[25651]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:22.209 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:21 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:22.209 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:21 smithi033 ceph-mon[25651]: from='osd.5 [v2:172.21.15.33:6808/4091940290,v1:172.21.15.33:6809/4091940290]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T21:47:22.209 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:21 smithi033 ceph-mon[25651]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T21:47:22.209 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:21 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:22.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:21 smithi007 ceph-mon[29541]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:22.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:21 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:22.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:21 smithi007 ceph-mon[29541]: from='osd.5 [v2:172.21.15.33:6808/4091940290,v1:172.21.15.33:6809/4091940290]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T21:47:22.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:21 smithi007 ceph-mon[29541]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T21:47:22.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:21 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:22.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:21 smithi007 ceph-mon[24826]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:22.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:21 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:22.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:21 smithi007 ceph-mon[24826]: from='osd.5 [v2:172.21.15.33:6808/4091940290,v1:172.21.15.33:6809/4091940290]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T21:47:22.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:21 smithi007 ceph-mon[24826]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-15T21:47:22.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:21 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:23.181 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:22 smithi033 ceph-mon[25651]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T21:47:23.182 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:22 smithi033 ceph-mon[25651]: osdmap e37: 6 total, 5 up, 6 in 2024-04-15T21:47:23.182 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:22 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:23.182 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:22 smithi033 ceph-mon[25651]: from='osd.5 [v2:172.21.15.33:6808/4091940290,v1:172.21.15.33:6809/4091940290]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:23.182 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:22 smithi033 ceph-mon[25651]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:23.182 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:47:22 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:47:22.941+0000 7f5ced43e700 -1 osd.5 0 waiting for initial osdmap 2024-04-15T21:47:23.182 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:47:22 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:47:22.946+0000 7f5ce65d2700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T21:47:23.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:22 smithi007 ceph-mon[24826]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T21:47:23.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:22 smithi007 ceph-mon[24826]: osdmap e37: 6 total, 5 up, 6 in 2024-04-15T21:47:23.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:22 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:23.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:22 smithi007 ceph-mon[24826]: from='osd.5 [v2:172.21.15.33:6808/4091940290,v1:172.21.15.33:6809/4091940290]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:23.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:22 smithi007 ceph-mon[24826]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:23.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:22 smithi007 ceph-mon[29541]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-15T21:47:23.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:22 smithi007 ceph-mon[29541]: osdmap e37: 6 total, 5 up, 6 in 2024-04-15T21:47:23.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:22 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:23.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:22 smithi007 ceph-mon[29541]: from='osd.5 [v2:172.21.15.33:6808/4091940290,v1:172.21.15.33:6809/4091940290]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:23.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:22 smithi007 ceph-mon[29541]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:24.082 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:23 smithi033 ceph-mon[25651]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:24.083 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:23 smithi033 ceph-mon[25651]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-15T21:47:24.083 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:23 smithi033 ceph-mon[25651]: osdmap e38: 6 total, 5 up, 6 in 2024-04-15T21:47:24.083 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:23 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:24.083 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:23 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:24.214 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-15T21:47:24.214 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-15T21:47:24.214 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-04-15T21:47:24.214 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-04-15T21:47:24.215 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123404 s, 850 MB/s 2024-04-15T21:47:24.215 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-04-15T21:47:24.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:23 smithi007 ceph-mon[24826]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:24.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:23 smithi007 ceph-mon[24826]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-15T21:47:24.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:23 smithi007 ceph-mon[24826]: osdmap e38: 6 total, 5 up, 6 in 2024-04-15T21:47:24.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:23 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:24.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:23 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:24.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:23 smithi007 ceph-mon[29541]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-15T21:47:24.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:23 smithi007 ceph-mon[29541]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-15T21:47:24.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:23 smithi007 ceph-mon[29541]: osdmap e38: 6 total, 5 up, 6 in 2024-04-15T21:47:24.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:23 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:24.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:23 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:24.964 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_2 2024-04-15T21:47:25.195 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:24 smithi033 ceph-mon[25651]: purged_snaps scrub starts 2024-04-15T21:47:25.195 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:24 smithi033 ceph-mon[25651]: purged_snaps scrub ok 2024-04-15T21:47:25.195 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:24 smithi033 ceph-mon[25651]: osd.5 [v2:172.21.15.33:6808/4091940290,v1:172.21.15.33:6809/4091940290] boot 2024-04-15T21:47:25.195 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:24 smithi033 ceph-mon[25651]: osdmap e39: 6 total, 6 up, 6 in 2024-04-15T21:47:25.195 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:24 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:25.195 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:24 smithi033 ceph-mon[25651]: osdmap e40: 6 total, 6 up, 6 in 2024-04-15T21:47:25.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:24 smithi007 ceph-mon[24826]: purged_snaps scrub starts 2024-04-15T21:47:25.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:24 smithi007 ceph-mon[24826]: purged_snaps scrub ok 2024-04-15T21:47:25.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:24 smithi007 ceph-mon[24826]: osd.5 [v2:172.21.15.33:6808/4091940290,v1:172.21.15.33:6809/4091940290] boot 2024-04-15T21:47:25.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:24 smithi007 ceph-mon[24826]: osdmap e39: 6 total, 6 up, 6 in 2024-04-15T21:47:25.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:24 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:25.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:24 smithi007 ceph-mon[24826]: osdmap e40: 6 total, 6 up, 6 in 2024-04-15T21:47:25.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:24 smithi007 ceph-mon[29541]: purged_snaps scrub starts 2024-04-15T21:47:25.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:24 smithi007 ceph-mon[29541]: purged_snaps scrub ok 2024-04-15T21:47:25.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:24 smithi007 ceph-mon[29541]: osd.5 [v2:172.21.15.33:6808/4091940290,v1:172.21.15.33:6809/4091940290] boot 2024-04-15T21:47:25.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:24 smithi007 ceph-mon[29541]: osdmap e39: 6 total, 6 up, 6 in 2024-04-15T21:47:25.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:24 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:47:25.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:24 smithi007 ceph-mon[29541]: osdmap e40: 6 total, 6 up, 6 in 2024-04-15T21:47:26.607 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:26 smithi033 ceph-mon[25651]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:26.607 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:26 smithi033 ceph-mon[25651]: osdmap e41: 6 total, 6 up, 6 in 2024-04-15T21:47:26.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:26 smithi007 ceph-mon[24826]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:26.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:26 smithi007 ceph-mon[24826]: osdmap e41: 6 total, 6 up, 6 in 2024-04-15T21:47:26.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:26 smithi007 ceph-mon[29541]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:26.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:26 smithi007 ceph-mon[29541]: osdmap e41: 6 total, 6 up, 6 in 2024-04-15T21:47:27.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:27 smithi033 ceph-mon[25651]: osdmap e42: 6 total, 6 up, 6 in 2024-04-15T21:47:27.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:27 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:27.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:27 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:27.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:27 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:27.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:27 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:27.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:27 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:27.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:27 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:47:27.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:27 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:27.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:27 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:47:27.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:27 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:27.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[24826]: osdmap e42: 6 total, 6 up, 6 in 2024-04-15T21:47:27.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:27.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:27.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:27.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:27.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:27.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:47:27.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:27.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:47:27.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:27.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[29541]: osdmap e42: 6 total, 6 up, 6 in 2024-04-15T21:47:27.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:27.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:27.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:27.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:27.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:27.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:47:27.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:27.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:47:27.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:27 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:28.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:28 smithi033 ceph-mon[25651]: Detected new or changed devices on smithi033 2024-04-15T21:47:28.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:28 smithi033 ceph-mon[25651]: Adjusting osd_memory_target on smithi033 to 8546M 2024-04-15T21:47:28.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:28 smithi033 ceph-mon[25651]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:28.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:28 smithi033 ceph-mon[25651]: osdmap e43: 6 total, 6 up, 6 in 2024-04-15T21:47:28.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:28 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:47:28.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:28 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:47:28.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:28 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:28.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[24826]: Detected new or changed devices on smithi033 2024-04-15T21:47:28.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[24826]: Adjusting osd_memory_target on smithi033 to 8546M 2024-04-15T21:47:28.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[24826]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:28.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[24826]: osdmap e43: 6 total, 6 up, 6 in 2024-04-15T21:47:28.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:47:28.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:47:28.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:28.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[29541]: Detected new or changed devices on smithi033 2024-04-15T21:47:28.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[29541]: Adjusting osd_memory_target on smithi033 to 8546M 2024-04-15T21:47:28.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[29541]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:28.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[29541]: osdmap e43: 6 total, 6 up, 6 in 2024-04-15T21:47:28.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:47:28.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:47:28.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:28 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:29.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:29 smithi033 ceph-mon[25651]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:47:29.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:29 smithi007 ceph-mon[24826]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:47:29.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:29 smithi007 ceph-mon[29541]: from='client.24278 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:47:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:30 smithi033 ceph-mon[25651]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:30 smithi033 ceph-mon[25651]: from='client.? 172.21.15.33:0/3755130472' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e9ca75f-ce5b-4fd8-ba69-c5c4175b1d2a"}]: dispatch 2024-04-15T21:47:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:30 smithi033 ceph-mon[25651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e9ca75f-ce5b-4fd8-ba69-c5c4175b1d2a"}]: dispatch 2024-04-15T21:47:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:30 smithi033 ceph-mon[25651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e9ca75f-ce5b-4fd8-ba69-c5c4175b1d2a"}]': finished 2024-04-15T21:47:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:30 smithi033 ceph-mon[25651]: osdmap e44: 7 total, 6 up, 7 in 2024-04-15T21:47:30.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:30 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:30.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:30 smithi007 ceph-mon[24826]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:30.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:30 smithi007 ceph-mon[24826]: from='client.? 172.21.15.33:0/3755130472' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e9ca75f-ce5b-4fd8-ba69-c5c4175b1d2a"}]: dispatch 2024-04-15T21:47:30.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:30 smithi007 ceph-mon[24826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e9ca75f-ce5b-4fd8-ba69-c5c4175b1d2a"}]: dispatch 2024-04-15T21:47:30.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:30 smithi007 ceph-mon[24826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e9ca75f-ce5b-4fd8-ba69-c5c4175b1d2a"}]': finished 2024-04-15T21:47:30.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:30 smithi007 ceph-mon[24826]: osdmap e44: 7 total, 6 up, 7 in 2024-04-15T21:47:30.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:30 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:30.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:30 smithi007 ceph-mon[29541]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:30.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:30 smithi007 ceph-mon[29541]: from='client.? 172.21.15.33:0/3755130472' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e9ca75f-ce5b-4fd8-ba69-c5c4175b1d2a"}]: dispatch 2024-04-15T21:47:30.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:30 smithi007 ceph-mon[29541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0e9ca75f-ce5b-4fd8-ba69-c5c4175b1d2a"}]: dispatch 2024-04-15T21:47:30.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:30 smithi007 ceph-mon[29541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0e9ca75f-ce5b-4fd8-ba69-c5c4175b1d2a"}]': finished 2024-04-15T21:47:30.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:30 smithi007 ceph-mon[29541]: osdmap e44: 7 total, 6 up, 7 in 2024-04-15T21:47:30.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:30 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:31.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:31 smithi033 ceph-mon[25651]: from='client.? 172.21.15.33:0/1708341700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:47:31.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:31 smithi007 ceph-mon[24826]: from='client.? 172.21.15.33:0/1708341700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:47:31.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:31 smithi007 ceph-mon[29541]: from='client.? 172.21.15.33:0/1708341700' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:47:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:32 smithi033 ceph-mon[25651]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:32.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:32 smithi007 ceph-mon[24826]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:32.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:32 smithi007 ceph-mon[29541]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:34.643 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:34 smithi033 ceph-mon[25651]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:34.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:34 smithi007 ceph-mon[24826]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:34.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:34 smithi007 ceph-mon[29541]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:36.736 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:36 smithi033 ceph-mon[25651]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:36.736 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:36 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-15T21:47:36.736 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:36 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:36.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:36 smithi007 ceph-mon[24826]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:36.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:36 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-15T21:47:36.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:36 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:36.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:36 smithi007 ceph-mon[29541]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:36.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:36 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-15T21:47:36.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:36 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:37.762 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:37 smithi033 ceph-mon[25651]: Deploying daemon osd.6 on smithi033 2024-04-15T21:47:37.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:37 smithi007 ceph-mon[24826]: Deploying daemon osd.6 on smithi033 2024-04-15T21:47:37.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:37 smithi007 ceph-mon[29541]: Deploying daemon osd.6 on smithi033 2024-04-15T21:47:38.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:38 smithi033 ceph-mon[25651]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:38.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:38 smithi007 ceph-mon[24826]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:38.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:38 smithi007 ceph-mon[29541]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:40.706 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:40 smithi033 ceph-mon[25651]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:40.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:40 smithi007 ceph-mon[24826]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:40.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:40 smithi007 ceph-mon[29541]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:42.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:42 smithi007 ceph-mon[24826]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:42.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:42 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:42.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:42 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:47:42.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:42 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:42.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:42 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:47:42.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:42 smithi007 ceph-mon[29541]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:42.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:42 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:42.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:42 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:47:42.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:42 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:42.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:42 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:47:42.669 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:42 smithi033 ceph-mon[25651]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:42.670 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:42 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:42.670 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:42 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:47:42.670 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:42 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:42.670 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:42 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:47:44.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:44 smithi007 ceph-mon[24826]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:44.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:44 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:44.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:44 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:44.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:44 smithi007 ceph-mon[29541]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:44.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:44 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:44.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:44 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:44.594 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:44 smithi033 ceph-mon[25651]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:44.594 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:44 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:44.594 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:44 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:44.626 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 6 on host 'smithi033' 2024-04-15T21:47:45.255 DEBUG:teuthology.orchestra.run.smithi033:osd.6> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.6.service 2024-04-15T21:47:45.257 INFO:tasks.cephadm:Deploying osd.7 on smithi033 with /dev/vg_nvme/lv_1... 2024-04-15T21:47:45.258 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-15T21:47:45.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:45 smithi033 ceph-mon[25651]: from='osd.6 [v2:172.21.15.33:6816/2157169070,v1:172.21.15.33:6817/2157169070]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T21:47:45.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:45 smithi033 ceph-mon[25651]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T21:47:45.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:45 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:45.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:45 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:47:45.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:45 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:45.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:45 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:47:45.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:45 smithi007 ceph-mon[24826]: from='osd.6 [v2:172.21.15.33:6816/2157169070,v1:172.21.15.33:6817/2157169070]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T21:47:45.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:45 smithi007 ceph-mon[24826]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T21:47:45.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:45 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:45.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:45 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:47:45.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:45 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:45.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:45 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:47:45.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:45 smithi007 ceph-mon[29541]: from='osd.6 [v2:172.21.15.33:6816/2157169070,v1:172.21.15.33:6817/2157169070]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T21:47:45.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:45 smithi007 ceph-mon[29541]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-15T21:47:45.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:45 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:45.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:45 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:47:45.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:45 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:45.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:45 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:47:46.766 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:46 smithi033 ceph-mon[25651]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:46.767 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:46 smithi033 ceph-mon[25651]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T21:47:46.767 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:46 smithi033 ceph-mon[25651]: osdmap e45: 7 total, 6 up, 7 in 2024-04-15T21:47:46.767 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:46 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:46.767 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:46 smithi033 ceph-mon[25651]: from='osd.6 [v2:172.21.15.33:6816/2157169070,v1:172.21.15.33:6817/2157169070]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:46.767 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:46 smithi033 ceph-mon[25651]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:46.767 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:47:46 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:47:46.423+0000 7ff9baebc700 -1 osd.6 0 waiting for initial osdmap 2024-04-15T21:47:46.767 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:47:46 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:47:46.431+0000 7ff9b204c700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T21:47:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:46 smithi007 ceph-mon[29541]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:46 smithi007 ceph-mon[29541]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T21:47:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:46 smithi007 ceph-mon[29541]: osdmap e45: 7 total, 6 up, 7 in 2024-04-15T21:47:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:46 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:46 smithi007 ceph-mon[29541]: from='osd.6 [v2:172.21.15.33:6816/2157169070,v1:172.21.15.33:6817/2157169070]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:46 smithi007 ceph-mon[29541]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:46.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:46 smithi007 ceph-mon[24826]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:46.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:46 smithi007 ceph-mon[24826]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-15T21:47:46.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:46 smithi007 ceph-mon[24826]: osdmap e45: 7 total, 6 up, 7 in 2024-04-15T21:47:46.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:46 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:46.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:46 smithi007 ceph-mon[24826]: from='osd.6 [v2:172.21.15.33:6816/2157169070,v1:172.21.15.33:6817/2157169070]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:46.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:46 smithi007 ceph-mon[24826]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:47:47.739 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:47 smithi033 ceph-mon[25651]: purged_snaps scrub starts 2024-04-15T21:47:47.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:47 smithi033 ceph-mon[25651]: purged_snaps scrub ok 2024-04-15T21:47:47.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:47 smithi033 ceph-mon[25651]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-15T21:47:47.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:47 smithi033 ceph-mon[25651]: osdmap e46: 7 total, 6 up, 7 in 2024-04-15T21:47:47.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:47 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:47.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:47 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:47.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:47 smithi033 ceph-mon[25651]: osd.6 [v2:172.21.15.33:6816/2157169070,v1:172.21.15.33:6817/2157169070] boot 2024-04-15T21:47:47.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:47 smithi033 ceph-mon[25651]: osdmap e47: 7 total, 7 up, 7 in 2024-04-15T21:47:47.740 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:47 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:47.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[24826]: purged_snaps scrub starts 2024-04-15T21:47:47.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[24826]: purged_snaps scrub ok 2024-04-15T21:47:47.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[24826]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-15T21:47:47.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[24826]: osdmap e46: 7 total, 6 up, 7 in 2024-04-15T21:47:47.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:47.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:47.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[24826]: osd.6 [v2:172.21.15.33:6816/2157169070,v1:172.21.15.33:6817/2157169070] boot 2024-04-15T21:47:47.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[24826]: osdmap e47: 7 total, 7 up, 7 in 2024-04-15T21:47:47.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:47.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[29541]: purged_snaps scrub starts 2024-04-15T21:47:47.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[29541]: purged_snaps scrub ok 2024-04-15T21:47:47.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[29541]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-15T21:47:47.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[29541]: osdmap e46: 7 total, 6 up, 7 in 2024-04-15T21:47:47.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:47.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:47.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[29541]: osd.6 [v2:172.21.15.33:6816/2157169070,v1:172.21.15.33:6817/2157169070] boot 2024-04-15T21:47:47.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[29541]: osdmap e47: 7 total, 7 up, 7 in 2024-04-15T21:47:47.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:47 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:47:48.636 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:48 smithi033 ceph-mon[25651]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:48.636 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:48 smithi033 ceph-mon[25651]: osdmap e48: 7 total, 7 up, 7 in 2024-04-15T21:47:48.636 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-15T21:47:48.636 INFO:teuthology.orchestra.run.smithi033.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-15T21:47:48.636 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10+0 records in 2024-04-15T21:47:48.636 INFO:teuthology.orchestra.run.smithi033.stderr:10+0 records out 2024-04-15T21:47:48.637 INFO:teuthology.orchestra.run.smithi033.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0119961 s, 874 MB/s 2024-04-15T21:47:48.637 INFO:teuthology.orchestra.run.smithi033.stderr:--> Zapping successful for: 2024-04-15T21:47:48.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:48 smithi007 ceph-mon[24826]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:48.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:48 smithi007 ceph-mon[24826]: osdmap e48: 7 total, 7 up, 7 in 2024-04-15T21:47:48.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:48 smithi007 ceph-mon[29541]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-15T21:47:48.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:48 smithi007 ceph-mon[29541]: osdmap e48: 7 total, 7 up, 7 in 2024-04-15T21:47:49.719 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch daemon add osd smithi033:vg_nvme/lv_1 2024-04-15T21:47:50.312 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:50 smithi033 ceph-mon[25651]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:47:50.312 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:50 smithi033 ceph-mon[25651]: Detected new or changed devices on smithi033 2024-04-15T21:47:50.312 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:50 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:50.312 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:50 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:50.313 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:50 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:50.313 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:50 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:50.313 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:50 smithi033 ceph-mon[25651]: Adjusting osd_memory_target on smithi033 to 5697M 2024-04-15T21:47:50.313 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:50 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:50.313 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:50 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:50.313 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:50 smithi033 ceph-mon[25651]: osdmap e49: 7 total, 7 up, 7 in 2024-04-15T21:47:50.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[29541]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:47:50.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[29541]: Detected new or changed devices on smithi033 2024-04-15T21:47:50.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:50.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:50.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:50.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:50.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[29541]: Adjusting osd_memory_target on smithi033 to 5697M 2024-04-15T21:47:50.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:50.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:50.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[29541]: osdmap e49: 7 total, 7 up, 7 in 2024-04-15T21:47:50.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[24826]: pgmap v146: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:47:50.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[24826]: Detected new or changed devices on smithi033 2024-04-15T21:47:50.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:50.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:50.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:50.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:47:50.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[24826]: Adjusting osd_memory_target on smithi033 to 5697M 2024-04-15T21:47:50.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:50.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:47:50.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:50 smithi007 ceph-mon[24826]: osdmap e49: 7 total, 7 up, 7 in 2024-04-15T21:47:51.586 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:51 smithi033 ceph-mon[25651]: osdmap e50: 7 total, 7 up, 7 in 2024-04-15T21:47:51.738 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:51 smithi007 ceph-mon[24826]: osdmap e50: 7 total, 7 up, 7 in 2024-04-15T21:47:51.739 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:51 smithi007 ceph-mon[29541]: osdmap e50: 7 total, 7 up, 7 in 2024-04-15T21:47:52.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:52 smithi033 ceph-mon[25651]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:47:52.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:52 smithi033 ceph-mon[25651]: osdmap e51: 7 total, 7 up, 7 in 2024-04-15T21:47:52.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:52 smithi033 ceph-mon[25651]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:47:52.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:47:52.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:47:52.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:52 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:52.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:52 smithi007 ceph-mon[24826]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:47:52.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:52 smithi007 ceph-mon[24826]: osdmap e51: 7 total, 7 up, 7 in 2024-04-15T21:47:52.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:52 smithi007 ceph-mon[24826]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:47:52.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:52 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:47:52.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:52 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:47:52.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:52 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:52.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:52 smithi007 ceph-mon[29541]: pgmap v149: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:47:52.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:52 smithi007 ceph-mon[29541]: osdmap e51: 7 total, 7 up, 7 in 2024-04-15T21:47:52.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:52 smithi007 ceph-mon[29541]: from='client.24305 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi033:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:47:52.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:52 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-15T21:47:52.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:52 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-15T21:47:52.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:52 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:47:54.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:54 smithi033 ceph-mon[25651]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-04-15T21:47:54.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:54 smithi033 ceph-mon[25651]: from='client.? 172.21.15.33:0/3297012570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b9473c1-b160-4b4d-b86f-0d678e76e9ed"}]: dispatch 2024-04-15T21:47:54.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:54 smithi033 ceph-mon[25651]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b9473c1-b160-4b4d-b86f-0d678e76e9ed"}]: dispatch 2024-04-15T21:47:54.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:54 smithi033 ceph-mon[25651]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b9473c1-b160-4b4d-b86f-0d678e76e9ed"}]': finished 2024-04-15T21:47:54.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:54 smithi033 ceph-mon[25651]: osdmap e52: 8 total, 7 up, 8 in 2024-04-15T21:47:54.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:54 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:47:54.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:54 smithi007 ceph-mon[24826]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-04-15T21:47:54.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:54 smithi007 ceph-mon[24826]: from='client.? 172.21.15.33:0/3297012570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b9473c1-b160-4b4d-b86f-0d678e76e9ed"}]: dispatch 2024-04-15T21:47:54.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:54 smithi007 ceph-mon[24826]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b9473c1-b160-4b4d-b86f-0d678e76e9ed"}]: dispatch 2024-04-15T21:47:54.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:54 smithi007 ceph-mon[24826]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b9473c1-b160-4b4d-b86f-0d678e76e9ed"}]': finished 2024-04-15T21:47:54.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:54 smithi007 ceph-mon[24826]: osdmap e52: 8 total, 7 up, 8 in 2024-04-15T21:47:54.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:54 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:47:54.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:54 smithi007 ceph-mon[29541]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-04-15T21:47:54.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:54 smithi007 ceph-mon[29541]: from='client.? 172.21.15.33:0/3297012570' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b9473c1-b160-4b4d-b86f-0d678e76e9ed"}]: dispatch 2024-04-15T21:47:54.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:54 smithi007 ceph-mon[29541]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2b9473c1-b160-4b4d-b86f-0d678e76e9ed"}]: dispatch 2024-04-15T21:47:54.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:54 smithi007 ceph-mon[29541]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2b9473c1-b160-4b4d-b86f-0d678e76e9ed"}]': finished 2024-04-15T21:47:54.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:54 smithi007 ceph-mon[29541]: osdmap e52: 8 total, 7 up, 8 in 2024-04-15T21:47:54.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:54 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:47:55.751 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:55 smithi033 ceph-mon[25651]: from='client.? 172.21.15.33:0/2345933380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:47:55.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:55 smithi007 ceph-mon[24826]: from='client.? 172.21.15.33:0/2345933380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:47:55.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:55 smithi007 ceph-mon[29541]: from='client.? 172.21.15.33:0/2345933380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-15T21:47:56.764 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:56 smithi033 ceph-mon[25651]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-15T21:47:56.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:56 smithi007 ceph-mon[24826]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-15T21:47:56.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:56 smithi007 ceph-mon[29541]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-04-15T21:47:58.726 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:47:58 smithi033 ceph-mon[25651]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-15T21:47:58.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:47:58 smithi007 ceph-mon[24826]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-15T21:47:58.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:47:58 smithi007 ceph-mon[29541]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-15T21:48:00.736 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:00 smithi033 ceph-mon[25651]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T21:48:00.736 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:00 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-15T21:48:00.736 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:00 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:00.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:00 smithi007 ceph-mon[24826]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T21:48:00.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:00 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-15T21:48:00.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:00 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:00.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:00 smithi007 ceph-mon[29541]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-15T21:48:00.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:00 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-15T21:48:00.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:00 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:01.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:01 smithi007 ceph-mon[24826]: Deploying daemon osd.7 on smithi033 2024-04-15T21:48:01.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:01 smithi007 ceph-mon[29541]: Deploying daemon osd.7 on smithi033 2024-04-15T21:48:01.727 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:01 smithi033 ceph-mon[25651]: Deploying daemon osd.7 on smithi033 2024-04-15T21:48:02.718 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:02 smithi033 ceph-mon[25651]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-15T21:48:02.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:02 smithi007 ceph-mon[24826]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-15T21:48:02.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:02 smithi007 ceph-mon[29541]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-15T21:48:04.580 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:04 smithi033 ceph-mon[25651]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:04.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:04 smithi007 ceph-mon[24826]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:04.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:04 smithi007 ceph-mon[29541]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:06.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:06 smithi033 ceph-mon[25651]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:06.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:06.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:06.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:06.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:06 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:48:06.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:06 smithi007 ceph-mon[29541]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:06.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:06.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:06.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:06.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:06 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:48:06.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:06 smithi007 ceph-mon[24826]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:06.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:06.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:06.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:06.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:06 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:48:08.477 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:08 smithi033 ceph-mon[25651]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:08.477 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:08 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:08.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:08 smithi007 ceph-mon[24826]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:08.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:08 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:08.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:08 smithi007 ceph-mon[29541]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:08.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:08 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:08.792 INFO:teuthology.orchestra.run.smithi033.stdout:Created osd(s) 7 on host 'smithi033' 2024-04-15T21:48:09.390 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:09 smithi033 ceph-mon[25651]: from='osd.7 [v2:172.21.15.33:6824/3140679968,v1:172.21.15.33:6825/3140679968]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T21:48:09.391 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:09 smithi033 ceph-mon[25651]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T21:48:09.391 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:09 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:09.391 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:09 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:09.391 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:09 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:09.391 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:09 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:09.391 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:09 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:48:09.412 DEBUG:teuthology.orchestra.run.smithi033:osd.7> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.7.service 2024-04-15T21:48:09.414 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-15T21:48:09.415 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd stat -f json 2024-04-15T21:48:09.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[24826]: from='osd.7 [v2:172.21.15.33:6824/3140679968,v1:172.21.15.33:6825/3140679968]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T21:48:09.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[24826]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T21:48:09.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:09.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:09.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:09.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:09.447 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:48:09.447 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[29541]: from='osd.7 [v2:172.21.15.33:6824/3140679968,v1:172.21.15.33:6825/3140679968]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T21:48:09.447 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[29541]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-15T21:48:09.447 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:09.447 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:09.448 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:09.448 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:09.448 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:09 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:48:10.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:10 smithi007 ceph-mon[24826]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:10.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:10 smithi007 ceph-mon[24826]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-15T21:48:10.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:10 smithi007 ceph-mon[24826]: osdmap e53: 8 total, 7 up, 8 in 2024-04-15T21:48:10.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:10 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:10.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:10 smithi007 ceph-mon[24826]: from='osd.7 [v2:172.21.15.33:6824/3140679968,v1:172.21.15.33:6825/3140679968]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:48:10.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:10 smithi007 ceph-mon[24826]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:48:10.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:10 smithi007 ceph-mon[29541]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:10.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:10 smithi007 ceph-mon[29541]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-15T21:48:10.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:10 smithi007 ceph-mon[29541]: osdmap e53: 8 total, 7 up, 8 in 2024-04-15T21:48:10.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:10 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:10.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:10 smithi007 ceph-mon[29541]: from='osd.7 [v2:172.21.15.33:6824/3140679968,v1:172.21.15.33:6825/3140679968]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:48:10.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:10 smithi007 ceph-mon[29541]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:48:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:10 smithi033 ceph-mon[25651]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:10 smithi033 ceph-mon[25651]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-15T21:48:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:10 smithi033 ceph-mon[25651]: osdmap e53: 8 total, 7 up, 8 in 2024-04-15T21:48:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:10 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:10 smithi033 ceph-mon[25651]: from='osd.7 [v2:172.21.15.33:6824/3140679968,v1:172.21.15.33:6825/3140679968]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:48:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:10 smithi033 ceph-mon[25651]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]: dispatch 2024-04-15T21:48:10.524 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:48:10 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:48:10.203+0000 7f3344e9c700 -1 osd.7 0 waiting for initial osdmap 2024-04-15T21:48:10.524 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:48:10 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:48:10.212+0000 7f333d82f700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-15T21:48:11.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:11 smithi007 ceph-mon[24826]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-15T21:48:11.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:11 smithi007 ceph-mon[24826]: osdmap e54: 8 total, 7 up, 8 in 2024-04-15T21:48:11.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:11 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:11.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:11 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:11.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:11 smithi007 ceph-mon[29541]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-15T21:48:11.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:11 smithi007 ceph-mon[29541]: osdmap e54: 8 total, 7 up, 8 in 2024-04-15T21:48:11.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:11 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:11.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:11 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:11.305 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:48:11.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:11 smithi033 ceph-mon[25651]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi033", "root=default"]}]': finished 2024-04-15T21:48:11.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:11 smithi033 ceph-mon[25651]: osdmap e54: 8 total, 7 up, 8 in 2024-04-15T21:48:11.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:11 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:11.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:11 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:11.907 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":55,"num_osds":8,"num_up_osds":8,"osd_up_since":1713217691,"num_in_osds":8,"osd_in_since":1713217673,"num_remapped_pgs":1} 2024-04-15T21:48:11.908 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T21:48:12.173 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:48:12.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:12 smithi033 ceph-mon[25651]: purged_snaps scrub starts 2024-04-15T21:48:12.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:12 smithi033 ceph-mon[25651]: purged_snaps scrub ok 2024-04-15T21:48:12.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:12 smithi033 ceph-mon[25651]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:12.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:12 smithi033 ceph-mon[25651]: osd.7 [v2:172.21.15.33:6824/3140679968,v1:172.21.15.33:6825/3140679968] boot 2024-04-15T21:48:12.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:12 smithi033 ceph-mon[25651]: osdmap e55: 8 total, 8 up, 8 in 2024-04-15T21:48:12.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:12 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:12.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:12 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1752688843' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-15T21:48:12.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:12 smithi033 ceph-mon[25651]: osdmap e56: 8 total, 8 up, 8 in 2024-04-15T21:48:12.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[24826]: purged_snaps scrub starts 2024-04-15T21:48:12.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[24826]: purged_snaps scrub ok 2024-04-15T21:48:12.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[24826]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:12.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[24826]: osd.7 [v2:172.21.15.33:6824/3140679968,v1:172.21.15.33:6825/3140679968] boot 2024-04-15T21:48:12.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[24826]: osdmap e55: 8 total, 8 up, 8 in 2024-04-15T21:48:12.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:12.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1752688843' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-15T21:48:12.515 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[24826]: osdmap e56: 8 total, 8 up, 8 in 2024-04-15T21:48:12.515 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[29541]: purged_snaps scrub starts 2024-04-15T21:48:12.515 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[29541]: purged_snaps scrub ok 2024-04-15T21:48:12.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[29541]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-15T21:48:12.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[29541]: osd.7 [v2:172.21.15.33:6824/3140679968,v1:172.21.15.33:6825/3140679968] boot 2024-04-15T21:48:12.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[29541]: osdmap e55: 8 total, 8 up, 8 in 2024-04-15T21:48:12.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:12.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1752688843' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-15T21:48:12.516 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:12 smithi007 ceph-mon[29541]: osdmap e56: 8 total, 8 up, 8 in 2024-04-15T21:48:13.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[24826]: Detected new or changed devices on smithi033 2024-04-15T21:48:13.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:13.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:48:13.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:48:13.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:48:13.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:48:13.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[24826]: Adjusting osd_memory_target on smithi033 to 4273M 2024-04-15T21:48:13.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:13.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:13.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:48:13.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:48:13.505 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[24826]: osdmap e57: 8 total, 8 up, 8 in 2024-04-15T21:48:13.505 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[29541]: Detected new or changed devices on smithi033 2024-04-15T21:48:13.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:13.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:48:13.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:48:13.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:48:13.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:48:13.506 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[29541]: Adjusting osd_memory_target on smithi033 to 4273M 2024-04-15T21:48:13.507 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:13.507 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:13.507 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:48:13.507 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:48:13.507 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:13 smithi007 ceph-mon[29541]: osdmap e57: 8 total, 8 up, 8 in 2024-04-15T21:48:13.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:13 smithi033 ceph-mon[25651]: Detected new or changed devices on smithi033 2024-04-15T21:48:13.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:13.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:48:13.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:48:13.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:48:13.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-15T21:48:13.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:13 smithi033 ceph-mon[25651]: Adjusting osd_memory_target on smithi033 to 4273M 2024-04-15T21:48:13.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:13.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:13.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:48:13.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:13 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:48:13.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:13 smithi033 ceph-mon[25651]: osdmap e57: 8 total, 8 up, 8 in 2024-04-15T21:48:13.938 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:48:13.939 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":57,"fsid":"18ed60a0-fb71-11ee-bc8f-c7b262605968","created":"2024-04-15T21:43:20.122319+0000","modified":"2024-04-15T21:48:13.202758+0000","last_up_change":"2024-04-15T21:48:11.196763+0000","last_in_change":"2024-04-15T21:47:53.879708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T21:46:18.889328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"57b79fc0-6f79-4eec-9331-1a2f7f45d0f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2067680297},{"type":"v1","addr":"172.21.15.7:6803","nonce":2067680297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2067680297},{"type":"v1","addr":"172.21.15.7:6805","nonce":2067680297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2067680297},{"type":"v1","addr":"172.21.15.7:6809","nonce":2067680297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2067680297},{"type":"v1","addr":"172.21.15.7:6807","nonce":2067680297}]},"public_addr":"172.21.15.7:6803/2067680297","cluster_addr":"172.21.15.7:6805/2067680297","heartbeat_back_addr":"172.21.15.7:6809/2067680297","heartbeat_front_addr":"172.21.15.7:6807/2067680297","state":["exists","up"]},{"osd":1,"uuid":"e9184457-c172-4ea2-a429-49cd5ce3a01a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1851194794},{"type":"v1","addr":"172.21.15.7:6811","nonce":1851194794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1851194794},{"type":"v1","addr":"172.21.15.7:6813","nonce":1851194794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":1851194794},{"type":"v1","addr":"172.21.15.7:6817","nonce":1851194794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1851194794},{"type":"v1","addr":"172.21.15.7:6815","nonce":1851194794}]},"public_addr":"172.21.15.7:6811/1851194794","cluster_addr":"172.21.15.7:6813/1851194794","heartbeat_back_addr":"172.21.15.7:6817/1851194794","heartbeat_front_addr":"172.21.15.7:6815/1851194794","state":["exists","up"]},{"osd":2,"uuid":"668b55eb-c95a-46d6-8e89-718b5d394d89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3569809948},{"type":"v1","addr":"172.21.15.7:6819","nonce":3569809948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3569809948},{"type":"v1","addr":"172.21.15.7:6821","nonce":3569809948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3569809948},{"type":"v1","addr":"172.21.15.7:6825","nonce":3569809948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3569809948},{"type":"v1","addr":"172.21.15.7:6823","nonce":3569809948}]},"public_addr":"172.21.15.7:6819/3569809948","cluster_addr":"172.21.15.7:6821/3569809948","heartbeat_back_addr":"172.21.15.7:6825/3569809948","heartbeat_front_addr":"172.21.15.7:6823/3569809948","state":["exists","up"]},{"osd":3,"uuid":"26df7960-b04a-41c8-8f00-29c0bd10806f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":2388023749},{"type":"v1","addr":"172.21.15.7:6827","nonce":2388023749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":2388023749},{"type":"v1","addr":"172.21.15.7:6829","nonce":2388023749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":2388023749},{"type":"v1","addr":"172.21.15.7:6833","nonce":2388023749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":2388023749},{"type":"v1","addr":"172.21.15.7:6831","nonce":2388023749}]},"public_addr":"172.21.15.7:6827/2388023749","cluster_addr":"172.21.15.7:6829/2388023749","heartbeat_back_addr":"172.21.15.7:6833/2388023749","heartbeat_front_addr":"172.21.15.7:6831/2388023749","state":["exists","up"]},{"osd":4,"uuid":"a3c02631-9af8-4210-81dc-78b3e09d8248","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":993063057},{"type":"v1","addr":"172.21.15.33:6801","nonce":993063057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":993063057},{"type":"v1","addr":"172.21.15.33:6803","nonce":993063057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":993063057},{"type":"v1","addr":"172.21.15.33:6807","nonce":993063057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":993063057},{"type":"v1","addr":"172.21.15.33:6805","nonce":993063057}]},"public_addr":"172.21.15.33:6801/993063057","cluster_addr":"172.21.15.33:6803/993063057","heartbeat_back_addr":"172.21.15.33:6807/993063057","heartbeat_front_addr":"172.21.15.33:6805/993063057","state":["exists","up"]},{"osd":5,"uuid":"ad528133-1de4-4e6d-af65-fd06b811054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":4091940290},{"type":"v1","addr":"172.21.15.33:6809","nonce":4091940290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":4091940290},{"type":"v1","addr":"172.21.15.33:6811","nonce":4091940290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":4091940290},{"type":"v1","addr":"172.21.15.33:6815","nonce":4091940290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":4091940290},{"type":"v1","addr":"172.21.15.33:6813","nonce":4091940290}]},"public_addr":"172.21.15.33:6809/4091940290","cluster_addr":"172.21.15.33:6811/4091940290","heartbeat_back_addr":"172.21.15.33:6815/4091940290","heartbeat_front_addr":"172.21.15.33:6813/4091940290","state":["exists","up"]},{"osd":6,"uuid":"0e9ca75f-ce5b-4fd8-ba69-c5c4175b1d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":2157169070},{"type":"v1","addr":"172.21.15.33:6817","nonce":2157169070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2157169070},{"type":"v1","addr":"172.21.15.33:6819","nonce":2157169070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2157169070},{"type":"v1","addr":"172.21.15.33:6823","nonce":2157169070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2157169070},{"type":"v1","addr":"172.21.15.33:6821","nonce":2157169070}]},"public_addr":"172.21.15.33:6817/2157169070","cluster_addr":"172.21.15.33:6819/2157169070","heartbeat_back_addr":"172.21.15.33:6823/2157169070","heartbeat_front_addr":"172.21.15.33:6821/2157169070","state":["exists","up"]},{"osd":7,"uuid":"2b9473c1-b160-4b4d-b86f-0d678e76e9ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3140679968},{"type":"v1","addr":"172.21.15.33:6825","nonce":3140679968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3140679968},{"type":"v1","addr":"172.21.15.33:6827","nonce":3140679968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3140679968},{"type":"v1","addr":"172.21.15.33:6831","nonce":3140679968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3140679968},{"type":"v1","addr":"172.21.15.33:6829","nonce":3140679968}]},"public_addr":"172.21.15.33:6825/3140679968","cluster_addr":"172.21.15.33:6827/3140679968","heartbeat_back_addr":"172.21.15.33:6831/3140679968","heartbeat_front_addr":"172.21.15.33:6829/3140679968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:45:29.564043+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:45:51.844504+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:46:14.651534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:46:37.296913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:46:59.191741+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:47:22.182751+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:47:45.451353+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:48:09.275228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[0,6]}],"primary_temp":[],"blocklist":{"172.21.15.7:0/2974958078":"2024-04-16T21:44:12.524953+0000","172.21.15.7:6801/1363484615":"2024-04-16T21:44:12.524953+0000","172.21.15.7:6800/1363484615":"2024-04-16T21:44:12.524953+0000","172.21.15.7:6800/764447257":"2024-04-16T21:43:50.252562+0000","172.21.15.7:6801/764447257":"2024-04-16T21:43:50.252562+0000","172.21.15.7:0/129134569":"2024-04-16T21:43:50.252562+0000","172.21.15.7:0/4220829240":"2024-04-16T21:43:50.252562+0000","172.21.15.7:0/2513010551":"2024-04-16T21:44:12.524953+0000","172.21.15.7:0/4079377254":"2024-04-16T21:44:12.524953+0000","172.21.15.7:0/3034024802":"2024-04-16T21:43:50.252562+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T21:48:14.492 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:14 smithi007 ceph-mon[24826]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:14.492 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:14 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1283559511' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T21:48:14.492 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:14 smithi007 ceph-mon[24826]: osdmap e58: 8 total, 8 up, 8 in 2024-04-15T21:48:14.492 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:14 smithi007 ceph-mon[29541]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:14.492 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:14 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1283559511' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T21:48:14.493 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:14 smithi007 ceph-mon[29541]: osdmap e58: 8 total, 8 up, 8 in 2024-04-15T21:48:14.517 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-15T21:46:18.889328+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-15T21:48:14.517 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-15T21:48:14.741 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:48:14.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:14 smithi033 ceph-mon[25651]: pgmap v166: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:14.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:14 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1283559511' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T21:48:14.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:14 smithi033 ceph-mon[25651]: osdmap e58: 8 total, 8 up, 8 in 2024-04-15T21:48:16.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:16 smithi007 ceph-mon[24826]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:16.219 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:16 smithi007 ceph-mon[24826]: osdmap e59: 8 total, 8 up, 8 in 2024-04-15T21:48:16.219 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:16 smithi007 ceph-mon[29541]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:16.219 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:16 smithi007 ceph-mon[29541]: osdmap e59: 8 total, 8 up, 8 in 2024-04-15T21:48:16.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:16 smithi033 ceph-mon[25651]: pgmap v169: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:16.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:16 smithi033 ceph-mon[25651]: osdmap e59: 8 total, 8 up, 8 in 2024-04-15T21:48:16.584 INFO:teuthology.orchestra.run.smithi007.stdout:pg_num: 1 2024-04-15T21:48:17.256 INFO:tasks.cephadm:Adding prometheus.a on smithi033 2024-04-15T21:48:17.256 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch apply prometheus '1;smithi033=a' 2024-04-15T21:48:17.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:17 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1710197064' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T21:48:17.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:17 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1710197064' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T21:48:17.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:17 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1710197064' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-15T21:48:18.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:18 smithi033 ceph-mon[25651]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-15T21:48:18.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:18 smithi007 ceph-mon[29541]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-15T21:48:18.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:18 smithi007 ceph-mon[24826]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-04-15T21:48:19.183 INFO:teuthology.orchestra.run.smithi033.stdout:Scheduled prometheus update... 2024-04-15T21:48:19.799 DEBUG:teuthology.orchestra.run.smithi033:prometheus.a> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@prometheus.a.service 2024-04-15T21:48:19.802 INFO:tasks.cephadm:Adding node-exporter.a on smithi007 2024-04-15T21:48:19.802 INFO:tasks.cephadm:Adding node-exporter.b on smithi033 2024-04-15T21:48:19.802 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch apply node-exporter '2;smithi007=a;smithi033=b' 2024-04-15T21:48:20.488 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ignoring --setuser ceph since I am not root 2024-04-15T21:48:20.488 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ignoring --setgroup ceph since I am not root 2024-04-15T21:48:20.488 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-mgr[26834]: -- 172.21.15.33:0/3541340050 <== mon.2 v2:172.21.15.33:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fc35148340 con 0x55fc35120c00 2024-04-15T21:48:20.488 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-mon[25651]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T21:48:20.488 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-mon[25651]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:48:20.489 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-mon[25651]: Saving service prometheus spec with placement smithi033=a;count:1 2024-04-15T21:48:20.489 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:20.489 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:20.489 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:20.489 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:48:20.489 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:20.489 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T21:48:20.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ignoring --setuser ceph since I am not root 2024-04-15T21:48:20.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ignoring --setgroup ceph since I am not root 2024-04-15T21:48:20.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mgr[25060]: -- 172.21.15.7:0/710704036 <== mon.0 v2:172.21.15.7:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5580ac034340 con 0x5580ac00cc00 2024-04-15T21:48:20.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[24826]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T21:48:20.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[24826]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:48:20.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[24826]: Saving service prometheus spec with placement smithi033=a;count:1 2024-04-15T21:48:20.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:20.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:20.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:20.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:48:20.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:20.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T21:48:20.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[29541]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-04-15T21:48:20.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[29541]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:48:20.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[29541]: Saving service prometheus spec with placement smithi033=a;count:1 2024-04-15T21:48:20.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:20.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:20.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:20.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:48:20.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' 2024-04-15T21:48:20.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-15T21:48:21.023 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:20.750+0000 7f524cfc2000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T21:48:21.023 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:20 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:20.843+0000 7f524cfc2000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T21:48:21.042 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:20.761+0000 7f56bc829000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-15T21:48:21.042 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:20 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:20.853+0000 7f56bc829000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-15T21:48:21.380 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:21 smithi033 ceph-mon[25651]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-15T21:48:21.380 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:21 smithi033 ceph-mon[25651]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-15T21:48:21.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:21 smithi007 ceph-mon[24826]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-15T21:48:21.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:21 smithi007 ceph-mon[24826]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-15T21:48:21.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:21.464+0000 7f56bc829000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T21:48:21.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:21 smithi007 ceph-mon[29541]: from='mgr.14150 172.21.15.7:0/2820277660' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-15T21:48:21.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:21 smithi007 ceph-mon[29541]: mgrmap e16: y(active, since 4m), standbys: x 2024-04-15T21:48:21.645 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:21 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:21.460+0000 7f524cfc2000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-15T21:48:21.645 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:21 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:21.554+0000 7f524cfc2000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T21:48:21.645 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:21 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:21.642+0000 7f524cfc2000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T21:48:21.878 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:21.556+0000 7f56bc829000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-15T21:48:21.879 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:21.642+0000 7f56bc829000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-15T21:48:21.879 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:21.794+0000 7f56bc829000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T21:48:21.990 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:21 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:21.795+0000 7f524cfc2000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-15T21:48:21.991 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:21 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:21.877+0000 7f524cfc2000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T21:48:21.991 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:21 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:21.988+0000 7f524cfc2000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T21:48:22.242 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:21.877+0000 7f56bc829000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-15T21:48:22.242 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:21.988+0000 7f56bc829000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-15T21:48:22.242 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:22 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:22.074+0000 7f56bc829000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T21:48:22.273 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:22 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:22.073+0000 7f524cfc2000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-15T21:48:22.274 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:22 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:22.238+0000 7f524cfc2000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T21:48:22.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:22 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:22.240+0000 7f56bc829000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-15T21:48:22.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:22 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:22.328+0000 7f56bc829000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T21:48:22.773 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:22 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:22.333+0000 7f524cfc2000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-15T21:48:23.705 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:23 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:23.377+0000 7f524cfc2000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T21:48:23.706 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:23 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:23.509+0000 7f524cfc2000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T21:48:23.706 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:23 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:23.703+0000 7f524cfc2000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T21:48:23.719 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:23 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:23.387+0000 7f56bc829000 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-15T21:48:23.719 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:23 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:23.521+0000 7f56bc829000 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-15T21:48:24.023 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:23 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:23.792+0000 7f524cfc2000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T21:48:24.023 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:23 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:23.930+0000 7f524cfc2000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T21:48:24.042 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:23 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:23.718+0000 7f56bc829000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-15T21:48:24.042 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:23 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:23.808+0000 7f56bc829000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-15T21:48:24.042 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:23 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:23.949+0000 7f56bc829000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-15T21:48:24.522 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:24 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:24.252+0000 7f524cfc2000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T21:48:24.523 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:24 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:24.477+0000 7f524cfc2000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T21:48:24.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:24 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:24.276+0000 7f56bc829000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-15T21:48:24.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:24 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:24.505+0000 7f56bc829000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-15T21:48:25.023 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:24 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:24.563+0000 7f524cfc2000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T21:48:25.023 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:24 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:24.737+0000 7f524cfc2000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T21:48:25.042 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:24 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:24.592+0000 7f56bc829000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-15T21:48:25.042 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:24 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:24.768+0000 7f56bc829000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-15T21:48:25.975 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:25 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:25.672+0000 7f524cfc2000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T21:48:26.025 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:25 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:25.718+0000 7f56bc829000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-15T21:48:26.085 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:26.024+0000 7f56bc829000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T21:48:26.228 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:25 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:25.972+0000 7f524cfc2000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-15T21:48:26.228 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:26.068+0000 7f524cfc2000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T21:48:26.228 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:26.222+0000 7f524cfc2000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T21:48:26.229 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: [15/Apr/2024:21:48:26] ENGINE Bus STARTING 2024-04-15T21:48:26.281 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:26.122+0000 7f56bc829000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-15T21:48:26.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-mon[25651]: Standby manager daemon x restarted 2024-04-15T21:48:26.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-mon[25651]: Standby manager daemon x started 2024-04-15T21:48:26.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-mon[25651]: from='mgr.? 172.21.15.33:0/3039035573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T21:48:26.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-mon[25651]: from='mgr.? 172.21.15.33:0/3039035573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T21:48:26.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-mon[25651]: from='mgr.? 172.21.15.33:0/3039035573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T21:48:26.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-mon[25651]: from='mgr.? 172.21.15.33:0/3039035573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T21:48:26.523 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: CherryPy Checker: 2024-04-15T21:48:26.523 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: The Application mounted at '' has an empty config. 2024-04-15T21:48:26.524 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: 2024-04-15T21:48:26.524 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: [15/Apr/2024:21:48:26] ENGINE Serving on http://:::9283 2024-04-15T21:48:26.524 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:26 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: [15/Apr/2024:21:48:26] ENGINE Bus STARTED 2024-04-15T21:48:26.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:26.280+0000 7f56bc829000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-15T21:48:26.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-mon[24826]: Standby manager daemon x restarted 2024-04-15T21:48:26.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-mon[24826]: Standby manager daemon x started 2024-04-15T21:48:26.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-mon[24826]: from='mgr.? 172.21.15.33:0/3039035573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T21:48:26.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-mon[24826]: from='mgr.? 172.21.15.33:0/3039035573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T21:48:26.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-mon[24826]: from='mgr.? 172.21.15.33:0/3039035573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T21:48:26.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-mon[24826]: from='mgr.? 172.21.15.33:0/3039035573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T21:48:26.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-mon[29541]: Standby manager daemon x restarted 2024-04-15T21:48:26.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-mon[29541]: Standby manager daemon x started 2024-04-15T21:48:26.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-mon[29541]: from='mgr.? 172.21.15.33:0/3039035573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-04-15T21:48:26.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-mon[29541]: from='mgr.? 172.21.15.33:0/3039035573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-15T21:48:26.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-mon[29541]: from='mgr.? 172.21.15.33:0/3039035573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-04-15T21:48:26.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:26 smithi007 ceph-mon[29541]: from='mgr.? 172.21.15.33:0/3039035573' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-15T21:48:27.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-mon[29541]: mgrmap e17: y(active, since 4m), standbys: x 2024-04-15T21:48:27.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-mon[29541]: Active manager daemon y restarted 2024-04-15T21:48:27.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-mon[29541]: Activating manager daemon y 2024-04-15T21:48:27.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-mon[29541]: osdmap e60: 8 total, 8 up, 8 in 2024-04-15T21:48:27.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-mon[24826]: mgrmap e17: y(active, since 4m), standbys: x 2024-04-15T21:48:27.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-mon[24826]: Active manager daemon y restarted 2024-04-15T21:48:27.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-mon[24826]: Activating manager daemon y 2024-04-15T21:48:27.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-mon[24826]: osdmap e60: 8 total, 8 up, 8 in 2024-04-15T21:48:27.543 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: [15/Apr/2024:21:48:27] ENGINE Bus STARTING 2024-04-15T21:48:27.690 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:27 smithi033 ceph-mon[25651]: mgrmap e17: y(active, since 4m), standbys: x 2024-04-15T21:48:27.690 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:27 smithi033 ceph-mon[25651]: Active manager daemon y restarted 2024-04-15T21:48:27.690 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:27 smithi033 ceph-mon[25651]: Activating manager daemon y 2024-04-15T21:48:27.690 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:27 smithi033 ceph-mon[25651]: osdmap e60: 8 total, 8 up, 8 in 2024-04-15T21:48:27.873 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: CherryPy Checker: 2024-04-15T21:48:27.873 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: The Application mounted at '' has an empty config. 2024-04-15T21:48:27.874 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:48:27.874 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: [15/Apr/2024:21:48:27] ENGINE Serving on http://:::9283 2024-04-15T21:48:27.874 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:27 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: [15/Apr/2024:21:48:27] ENGINE Bus STARTED 2024-04-15T21:48:28.313 INFO:teuthology.orchestra.run.smithi033.stdout:Scheduled node-exporter update... 2024-04-15T21:48:28.542 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: [15/Apr/2024:21:48:28] ENGINE Bus STARTING 2024-04-15T21:48:28.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: mgrmap e18: y(active, starting, since 1.00463s), standbys: x 2024-04-15T21:48:28.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:48:28.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:48:28.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:48:28.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T21:48:28.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T21:48:28.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:48:28.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:48:28.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:48:28.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:48:28.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:48:28.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:48:28.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: Manager daemon y is now available 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:48:28.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:48:28.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: mgrmap e18: y(active, starting, since 1.00463s), standbys: x 2024-04-15T21:48:28.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:48:28.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:48:28.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:48:28.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T21:48:28.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T21:48:28.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:48:28.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:48:28.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:48:28.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:48:28.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:48:28.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:48:28.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: Manager daemon y is now available 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:48:28.546 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:48:28.633 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: mgrmap e18: y(active, starting, since 1.00463s), standbys: x 2024-04-15T21:48:28.633 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-15T21:48:28.633 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-15T21:48:28.634 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-15T21:48:28.635 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: Manager daemon y is now available 2024-04-15T21:48:28.635 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:28.635 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:48:28.635 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:48:28.635 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:48:28.635 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:28.635 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:48:28.635 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:48:28.635 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:48:28.635 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:28 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:48:28.960 DEBUG:teuthology.orchestra.run.smithi007:node-exporter.a> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@node-exporter.a.service 2024-04-15T21:48:28.962 DEBUG:teuthology.orchestra.run.smithi033:node-exporter.b> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@node-exporter.b.service 2024-04-15T21:48:28.965 INFO:tasks.cephadm:Adding alertmanager.a on smithi007 2024-04-15T21:48:28.965 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch apply alertmanager '1;smithi007=a' 2024-04-15T21:48:28.973 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: [15/Apr/2024:21:48:28] ENGINE Serving on https://172.21.15.7:7150 2024-04-15T21:48:28.973 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:28 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: [15/Apr/2024:21:48:28] ENGINE Bus STARTED 2024-04-15T21:48:29.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[24826]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-15T21:48:29.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[24826]: Saving service node-exporter spec with placement smithi007=a;smithi033=b;count:2 2024-04-15T21:48:29.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:29.565 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[24826]: [15/Apr/2024:21:48:28] ENGINE Bus STARTING 2024-04-15T21:48:29.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[24826]: [15/Apr/2024:21:48:28] ENGINE Serving on https://172.21.15.7:7150 2024-04-15T21:48:29.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[24826]: [15/Apr/2024:21:48:28] ENGINE Bus STARTED 2024-04-15T21:48:29.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:29.566 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:29.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[29541]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-15T21:48:29.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[29541]: Saving service node-exporter spec with placement smithi007=a;smithi033=b;count:2 2024-04-15T21:48:29.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:29.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[29541]: [15/Apr/2024:21:48:28] ENGINE Bus STARTING 2024-04-15T21:48:29.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[29541]: [15/Apr/2024:21:48:28] ENGINE Serving on https://172.21.15.7:7150 2024-04-15T21:48:29.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[29541]: [15/Apr/2024:21:48:28] ENGINE Bus STARTED 2024-04-15T21:48:29.566 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:29.567 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:29 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:29.599 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:29 smithi033 ceph-mon[25651]: mgrmap e19: y(active, since 2s), standbys: x 2024-04-15T21:48:29.599 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:29 smithi033 ceph-mon[25651]: Saving service node-exporter spec with placement smithi007=a;smithi033=b;count:2 2024-04-15T21:48:29.599 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:29 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:29.599 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:29 smithi033 ceph-mon[25651]: [15/Apr/2024:21:48:28] ENGINE Bus STARTING 2024-04-15T21:48:29.599 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:29 smithi033 ceph-mon[25651]: [15/Apr/2024:21:48:28] ENGINE Serving on https://172.21.15.7:7150 2024-04-15T21:48:29.600 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:29 smithi033 ceph-mon[25651]: [15/Apr/2024:21:48:28] ENGINE Bus STARTED 2024-04-15T21:48:29.600 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:29 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:29.600 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:29 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:30.696 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:30 smithi033 ceph-mon[25651]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:30.696 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:30 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:30.696 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:30 smithi033 ceph-mon[25651]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-15T21:48:30.696 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:30 smithi033 ceph-mon[25651]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-15T21:48:30.696 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:30 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:30.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:30 smithi007 ceph-mon[24826]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:30.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:30 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:30.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:30 smithi007 ceph-mon[24826]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-15T21:48:30.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:30 smithi007 ceph-mon[24826]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-15T21:48:30.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:30 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:30.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:30 smithi007 ceph-mon[29541]: pgmap v4: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:30.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:30 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:30.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:30 smithi007 ceph-mon[29541]: Updating smithi007:/etc/ceph/ceph.conf 2024-04-15T21:48:30.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:30 smithi007 ceph-mon[29541]: Updating smithi007:/etc/ceph/ceph.client.admin.keyring 2024-04-15T21:48:30.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:30 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:31.127 INFO:teuthology.orchestra.run.smithi033.stdout:Scheduled alertmanager update... 2024-04-15T21:48:31.751 DEBUG:teuthology.orchestra.run.smithi007:alertmanager.a> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@alertmanager.a.service 2024-04-15T21:48:31.754 INFO:tasks.cephadm:Adding grafana.a on smithi033 2024-04-15T21:48:31.754 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph orch apply grafana '1;smithi033=a' 2024-04-15T21:48:31.984 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:31 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:31.984 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:31 smithi033 ceph-mon[25651]: from='client.24395 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:48:31.984 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:31 smithi033 ceph-mon[25651]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-04-15T21:48:31.984 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:31 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:31.984 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:31 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:32.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:31 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:32.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:31 smithi007 ceph-mon[24826]: from='client.24395 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:48:32.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:31 smithi007 ceph-mon[24826]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-04-15T21:48:32.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:31 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:32.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:31 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:32.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:31 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:32.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:31 smithi007 ceph-mon[29541]: from='client.24395 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi007=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:48:32.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:31 smithi007 ceph-mon[29541]: Saving service alertmanager spec with placement smithi007=a;count:1 2024-04-15T21:48:32.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:31 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:32.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:31 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:33.218 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:33 smithi033 ceph-mon[25651]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:33.218 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:33 smithi033 ceph-mon[25651]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-15T21:48:33.218 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:33 smithi033 ceph-mon[25651]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-04-15T21:48:33.218 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:33 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:33.218 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:33 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:33.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:33 smithi007 ceph-mon[24826]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:33.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:33 smithi007 ceph-mon[24826]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-15T21:48:33.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:33 smithi007 ceph-mon[24826]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-04-15T21:48:33.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:33 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:33.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:33 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:33.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:33 smithi007 ceph-mon[29541]: pgmap v5: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:33.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:33 smithi007 ceph-mon[29541]: Updating smithi033:/etc/ceph/ceph.conf 2024-04-15T21:48:33.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:33 smithi007 ceph-mon[29541]: Updating smithi033:/etc/ceph/ceph.client.admin.keyring 2024-04-15T21:48:33.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:33 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:33.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:33 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:33.915 INFO:teuthology.orchestra.run.smithi033.stdout:Scheduled grafana update... 2024-04-15T21:48:34.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:34 smithi033 ceph-mon[25651]: Deploying daemon prometheus.a on smithi033 2024-04-15T21:48:34.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:34 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:34.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:34 smithi007 ceph-mon[24826]: Deploying daemon prometheus.a on smithi033 2024-04-15T21:48:34.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:34 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:34.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:34 smithi007 ceph-mon[29541]: Deploying daemon prometheus.a on smithi033 2024-04-15T21:48:34.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:34 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:34.895 DEBUG:teuthology.orchestra.run.smithi033:grafana.a> sudo journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@grafana.a.service 2024-04-15T21:48:34.897 INFO:tasks.cephadm:Setting up client nodes... 2024-04-15T21:48:34.900 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-15T21:48:35.221 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:35 smithi007 ceph-mon[24826]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:35.221 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:35 smithi007 ceph-mon[24826]: from='client.24401 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:48:35.221 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:35 smithi007 ceph-mon[24826]: Saving service grafana spec with placement smithi033=a;count:1 2024-04-15T21:48:35.221 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:35 smithi007 ceph-mon[29541]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:35.221 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:35 smithi007 ceph-mon[29541]: from='client.24401 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:48:35.221 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:35 smithi007 ceph-mon[29541]: Saving service grafana spec with placement smithi033=a;count:1 2024-04-15T21:48:35.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:35 smithi033 ceph-mon[25651]: pgmap v6: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:35.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:35 smithi033 ceph-mon[25651]: from='client.24401 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi033=a", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:48:35.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:35 smithi033 ceph-mon[25651]: Saving service grafana spec with placement smithi033=a;count:1 2024-04-15T21:48:36.927 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-04-15T21:48:36.927 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQC0oB1mJJz+NhAA6IesfNHKPt0JCeLSfiQ2LA== 2024-04-15T21:48:37.222 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:37 smithi007 ceph-mon[24826]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:37.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:37 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/216746459' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T21:48:37.223 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:37 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/216746459' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T21:48:37.223 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:37 smithi007 ceph-mon[29541]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:37.223 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:37 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/216746459' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T21:48:37.223 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:37 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/216746459' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T21:48:37.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:37 smithi033 ceph-mon[25651]: pgmap v7: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:37.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:37 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/216746459' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T21:48:37.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:37 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/216746459' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T21:48:37.592 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-04-15T21:48:37.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-15T21:48:37.592 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-15T21:48:37.639 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-15T21:48:39.482 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:39 smithi033 ceph-mon[25651]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:39.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:39 smithi007 ceph-mon[24826]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:39.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:39 smithi007 ceph-mon[29541]: pgmap v8: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:41.363 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:41 smithi033 ceph-mon[25651]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:41.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:41 smithi007 ceph-mon[24826]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:41.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:41 smithi007 ceph-mon[29541]: pgmap v9: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:42.553 INFO:teuthology.orchestra.run.smithi033.stdout:[client.1] 2024-04-15T21:48:42.554 INFO:teuthology.orchestra.run.smithi033.stdout: key = AQC6oB1mtJOqIBAAwaeXdZ9QN48qKdJk8weHQQ== 2024-04-15T21:48:43.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:43 smithi033 ceph-mon[25651]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:43.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:43 smithi033 ceph-mon[25651]: from='client.? 172.21.15.33:0/2455493961' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T21:48:43.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:43 smithi033 ceph-mon[25651]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T21:48:43.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:43 smithi033 ceph-mon[25651]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T21:48:43.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:43 smithi007 ceph-mon[24826]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:43.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:43 smithi007 ceph-mon[24826]: from='client.? 172.21.15.33:0/2455493961' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T21:48:43.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:43 smithi007 ceph-mon[24826]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T21:48:43.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:43 smithi007 ceph-mon[24826]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T21:48:43.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:43 smithi007 ceph-mon[29541]: pgmap v10: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:43.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:43 smithi007 ceph-mon[29541]: from='client.? 172.21.15.33:0/2455493961' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T21:48:43.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:43 smithi007 ceph-mon[29541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-15T21:48:43.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:43 smithi007 ceph-mon[29541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-15T21:48:44.099 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 systemd[1]: Starting Ceph prometheus.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:48:44.363 DEBUG:teuthology.orchestra.run.smithi033:> set -ex 2024-04-15T21:48:44.364 DEBUG:teuthology.orchestra.run.smithi033:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-04-15T21:48:44.364 DEBUG:teuthology.orchestra.run.smithi033:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-04-15T21:48:44.409 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-15T21:48:44.410 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-15T21:48:44.410 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph mgr dump --format=json 2024-04-15T21:48:44.639 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:48:44.683 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 podman[40781]: 2024-04-15 21:48:44.58217672 +0000 UTC m=+0.102926111 container create 58d4b7cad8f16082714994e87ae2d80de1b132649ee31a67c0797bb3b6103619 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T21:48:44.683 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 podman[40781]: 2024-04-15 21:48:44.498025071 +0000 UTC m=+0.018774462 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-15T21:48:45.148 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 podman[40781]: 2024-04-15 21:48:44.792674416 +0000 UTC m=+0.313423805 container init 58d4b7cad8f16082714994e87ae2d80de1b132649ee31a67c0797bb3b6103619 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T21:48:45.148 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 podman[40781]: 2024-04-15 21:48:44.795254204 +0000 UTC m=+0.316003601 container start 58d4b7cad8f16082714994e87ae2d80de1b132649ee31a67c0797bb3b6103619 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T21:48:45.149 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.818Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-15T21:48:45.149 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.818Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-15T21:48:45.149 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.818Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-15T21:48:45.149 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.818Z caller=main.go:518 level=info host_details="(Linux 5.14.0-438.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Apr 12 13:55:21 UTC 2024 x86_64 smithi033 (none))" 2024-04-15T21:48:45.149 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.818Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-15T21:48:45.149 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.818Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-15T21:48:45.149 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.819Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-15T21:48:45.149 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.820Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-15T21:48:45.149 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.822Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-15T21:48:45.149 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.823Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-15T21:48:45.149 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.823Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.574µs 2024-04-15T21:48:45.149 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.823Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-15T21:48:45.150 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.823Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-04-15T21:48:45.150 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.823Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=20.273µs wal_replay_duration=128.143µs total_replay_duration=161.956µs 2024-04-15T21:48:45.150 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.824Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-15T21:48:45.150 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.824Z caller=main.go:947 level=info msg="TSDB started" 2024-04-15T21:48:45.150 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.824Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-15T21:48:45.150 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.840Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.010316ms db_storage=661ns remote_storage=1.291µs web_handler=198ns query_engine=567ns scrape=433.927µs scrape_sd=15.504µs notify=869ns notify_sd=1.441µs rules=15.240294ms 2024-04-15T21:48:45.150 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:48:44.840Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-15T21:48:45.150 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 bash[40781]: 58d4b7cad8f16082714994e87ae2d80de1b132649ee31a67c0797bb3b6103619 2024-04-15T21:48:45.150 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:48:44 smithi033 systemd[1]: Started Ceph prometheus.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:48:45.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:45 smithi007 ceph-mon[29541]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:45.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:45 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:45.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:45 smithi007 ceph-mon[24826]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:45.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:45 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:45.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:45 smithi033 ceph-mon[25651]: pgmap v11: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:45.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:45 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:46.410 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:46 smithi007 ceph-mon[24826]: Deploying daemon node-exporter.a on smithi007 2024-04-15T21:48:46.411 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:46 smithi007 ceph-mon[29541]: Deploying daemon node-exporter.a on smithi007 2024-04-15T21:48:46.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:46 smithi033 ceph-mon[25651]: Deploying daemon node-exporter.a on smithi007 2024-04-15T21:48:46.646 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:48:47.332 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":19,"active_gid":24358,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6800","nonce":1218414668},{"type":"v1","addr":"172.21.15.7:6801","nonce":1218414668}]},"active_addr":"172.21.15.7:6801/1218414668","active_change":"2024-04-15T21:48:26.282900+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":24353,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.7:8443/","prometheus":"http://172.21.15.7:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":60,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":2688359073}]},{"addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":938570794}]},{"addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":111228037}]},{"addrvec":[{"type":"v2","addr":"172.21.15.7:0","nonce":1357040679}]}]}} 2024-04-15T21:48:47.335 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-15T21:48:47.335 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-15T21:48:47.336 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T21:48:47.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:47 smithi033 ceph-mon[25651]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:47.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:47 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/4291945356' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-15T21:48:47.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:47 smithi007 ceph-mon[24826]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:47.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:47 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/4291945356' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-15T21:48:47.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:47 smithi007 ceph-mon[29541]: pgmap v12: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:47.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:47 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/4291945356' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-15T21:48:47.568 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:48:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:48 smithi007 ceph-mon[24826]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:48.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:48 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:48.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:48 smithi007 ceph-mon[29541]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:48.732 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:48 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:48.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:48 smithi033 ceph-mon[25651]: pgmap v13: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:48.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:48 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:49.449 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:49 smithi007 bash[44312]: Writing manifest to image destination 2024-04-15T21:48:49.450 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:49 smithi007 podman[44312]: 2024-04-15 21:48:49.354194893 +0000 UTC m=+2.931525435 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-15T21:48:50.102 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:49 smithi007 podman[44312]: 2024-04-15 21:48:49.796371017 +0000 UTC m=+3.373701556 container create 49d5ffb0b3d0bf83929da50acc402be5da2e113b2b8e528349e7237ca5e48799 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-15T21:48:50.102 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 podman[44312]: 2024-04-15 21:48:50.024570369 +0000 UTC m=+3.601900911 container init 49d5ffb0b3d0bf83929da50acc402be5da2e113b2b8e528349e7237ca5e48799 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-15T21:48:50.102 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 podman[44312]: 2024-04-15 21:48:50.027948082 +0000 UTC m=+3.605278631 container start 49d5ffb0b3d0bf83929da50acc402be5da2e113b2b8e528349e7237ca5e48799 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-15T21:48:50.102 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.032Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-15T21:48:50.102 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.032Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-15T21:48:50.102 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.033Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-15T21:48:50.102 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.033Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-15T21:48:50.102 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.033Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-15T21:48:50.102 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.033Z caller=node_exporter.go:115 level=info collector=arp 2024-04-15T21:48:50.102 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.033Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-15T21:48:50.102 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.033Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-15T21:48:50.103 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.033Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-15T21:48:50.103 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.033Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-15T21:48:50.103 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.033Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-15T21:48:50.103 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.033Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-15T21:48:50.103 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.033Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-15T21:48:50.103 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.033Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-15T21:48:50.103 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=edac 2024-04-15T21:48:50.103 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-15T21:48:50.103 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-15T21:48:50.104 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=os 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=stat 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=time 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=uname 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-15T21:48:50.105 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-15T21:48:50.106 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-15T21:48:50.106 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a[44488]: ts=2024-04-15T21:48:50.034Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-15T21:48:50.106 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 bash[44312]: 49d5ffb0b3d0bf83929da50acc402be5da2e113b2b8e528349e7237ca5e48799 2024-04-15T21:48:50.403 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:48:50 smithi007 systemd[1]: Started Ceph node-exporter.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:48:50.694 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:48:50.695 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":60,"fsid":"18ed60a0-fb71-11ee-bc8f-c7b262605968","created":"2024-04-15T21:43:20.122319+0000","modified":"2024-04-15T21:48:26.282785+0000","last_up_change":"2024-04-15T21:48:11.196763+0000","last_in_change":"2024-04-15T21:47:53.879708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T21:46:18.889328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"57b79fc0-6f79-4eec-9331-1a2f7f45d0f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2067680297},{"type":"v1","addr":"172.21.15.7:6803","nonce":2067680297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2067680297},{"type":"v1","addr":"172.21.15.7:6805","nonce":2067680297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2067680297},{"type":"v1","addr":"172.21.15.7:6809","nonce":2067680297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2067680297},{"type":"v1","addr":"172.21.15.7:6807","nonce":2067680297}]},"public_addr":"172.21.15.7:6803/2067680297","cluster_addr":"172.21.15.7:6805/2067680297","heartbeat_back_addr":"172.21.15.7:6809/2067680297","heartbeat_front_addr":"172.21.15.7:6807/2067680297","state":["exists","up"]},{"osd":1,"uuid":"e9184457-c172-4ea2-a429-49cd5ce3a01a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1851194794},{"type":"v1","addr":"172.21.15.7:6811","nonce":1851194794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1851194794},{"type":"v1","addr":"172.21.15.7:6813","nonce":1851194794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":1851194794},{"type":"v1","addr":"172.21.15.7:6817","nonce":1851194794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1851194794},{"type":"v1","addr":"172.21.15.7:6815","nonce":1851194794}]},"public_addr":"172.21.15.7:6811/1851194794","cluster_addr":"172.21.15.7:6813/1851194794","heartbeat_back_addr":"172.21.15.7:6817/1851194794","heartbeat_front_addr":"172.21.15.7:6815/1851194794","state":["exists","up"]},{"osd":2,"uuid":"668b55eb-c95a-46d6-8e89-718b5d394d89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3569809948},{"type":"v1","addr":"172.21.15.7:6819","nonce":3569809948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3569809948},{"type":"v1","addr":"172.21.15.7:6821","nonce":3569809948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3569809948},{"type":"v1","addr":"172.21.15.7:6825","nonce":3569809948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3569809948},{"type":"v1","addr":"172.21.15.7:6823","nonce":3569809948}]},"public_addr":"172.21.15.7:6819/3569809948","cluster_addr":"172.21.15.7:6821/3569809948","heartbeat_back_addr":"172.21.15.7:6825/3569809948","heartbeat_front_addr":"172.21.15.7:6823/3569809948","state":["exists","up"]},{"osd":3,"uuid":"26df7960-b04a-41c8-8f00-29c0bd10806f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":2388023749},{"type":"v1","addr":"172.21.15.7:6827","nonce":2388023749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":2388023749},{"type":"v1","addr":"172.21.15.7:6829","nonce":2388023749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":2388023749},{"type":"v1","addr":"172.21.15.7:6833","nonce":2388023749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":2388023749},{"type":"v1","addr":"172.21.15.7:6831","nonce":2388023749}]},"public_addr":"172.21.15.7:6827/2388023749","cluster_addr":"172.21.15.7:6829/2388023749","heartbeat_back_addr":"172.21.15.7:6833/2388023749","heartbeat_front_addr":"172.21.15.7:6831/2388023749","state":["exists","up"]},{"osd":4,"uuid":"a3c02631-9af8-4210-81dc-78b3e09d8248","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":993063057},{"type":"v1","addr":"172.21.15.33:6801","nonce":993063057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":993063057},{"type":"v1","addr":"172.21.15.33:6803","nonce":993063057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":993063057},{"type":"v1","addr":"172.21.15.33:6807","nonce":993063057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":993063057},{"type":"v1","addr":"172.21.15.33:6805","nonce":993063057}]},"public_addr":"172.21.15.33:6801/993063057","cluster_addr":"172.21.15.33:6803/993063057","heartbeat_back_addr":"172.21.15.33:6807/993063057","heartbeat_front_addr":"172.21.15.33:6805/993063057","state":["exists","up"]},{"osd":5,"uuid":"ad528133-1de4-4e6d-af65-fd06b811054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":4091940290},{"type":"v1","addr":"172.21.15.33:6809","nonce":4091940290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":4091940290},{"type":"v1","addr":"172.21.15.33:6811","nonce":4091940290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":4091940290},{"type":"v1","addr":"172.21.15.33:6815","nonce":4091940290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":4091940290},{"type":"v1","addr":"172.21.15.33:6813","nonce":4091940290}]},"public_addr":"172.21.15.33:6809/4091940290","cluster_addr":"172.21.15.33:6811/4091940290","heartbeat_back_addr":"172.21.15.33:6815/4091940290","heartbeat_front_addr":"172.21.15.33:6813/4091940290","state":["exists","up"]},{"osd":6,"uuid":"0e9ca75f-ce5b-4fd8-ba69-c5c4175b1d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":2157169070},{"type":"v1","addr":"172.21.15.33:6817","nonce":2157169070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2157169070},{"type":"v1","addr":"172.21.15.33:6819","nonce":2157169070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2157169070},{"type":"v1","addr":"172.21.15.33:6823","nonce":2157169070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2157169070},{"type":"v1","addr":"172.21.15.33:6821","nonce":2157169070}]},"public_addr":"172.21.15.33:6817/2157169070","cluster_addr":"172.21.15.33:6819/2157169070","heartbeat_back_addr":"172.21.15.33:6823/2157169070","heartbeat_front_addr":"172.21.15.33:6821/2157169070","state":["exists","up"]},{"osd":7,"uuid":"2b9473c1-b160-4b4d-b86f-0d678e76e9ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3140679968},{"type":"v1","addr":"172.21.15.33:6825","nonce":3140679968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3140679968},{"type":"v1","addr":"172.21.15.33:6827","nonce":3140679968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3140679968},{"type":"v1","addr":"172.21.15.33:6831","nonce":3140679968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3140679968},{"type":"v1","addr":"172.21.15.33:6829","nonce":3140679968}]},"public_addr":"172.21.15.33:6825/3140679968","cluster_addr":"172.21.15.33:6827/3140679968","heartbeat_back_addr":"172.21.15.33:6831/3140679968","heartbeat_front_addr":"172.21.15.33:6829/3140679968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:45:29.564043+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:45:51.844504+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:46:14.651534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:46:37.296913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:46:59.191741+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:47:22.182751+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:47:45.451353+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:48:09.275228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3332417937":"2024-04-16T21:48:26.282752+0000","172.21.15.7:0/903497089":"2024-04-16T21:48:26.282752+0000","172.21.15.7:6801/1416545093":"2024-04-16T21:48:26.282752+0000","172.21.15.7:0/2927327763":"2024-04-16T21:48:26.282752+0000","172.21.15.7:0/2974958078":"2024-04-16T21:44:12.524953+0000","172.21.15.7:6801/1363484615":"2024-04-16T21:44:12.524953+0000","172.21.15.7:6800/1363484615":"2024-04-16T21:44:12.524953+0000","172.21.15.7:6800/764447257":"2024-04-16T21:43:50.252562+0000","172.21.15.7:6801/764447257":"2024-04-16T21:43:50.252562+0000","172.21.15.7:0/129134569":"2024-04-16T21:43:50.252562+0000","172.21.15.7:0/1756042719":"2024-04-16T21:48:26.282752+0000","172.21.15.7:6800/1416545093":"2024-04-16T21:48:26.282752+0000","172.21.15.7:0/4220829240":"2024-04-16T21:43:50.252562+0000","172.21.15.7:0/2513010551":"2024-04-16T21:44:12.524953+0000","172.21.15.7:0/4079377254":"2024-04-16T21:44:12.524953+0000","172.21.15.7:0/3034024802":"2024-04-16T21:43:50.252562+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T21:48:51.174 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:51 smithi033 bash[41021]: Trying to pull quay.io/prometheus/node-exporter:v1.3.1... 2024-04-15T21:48:51.282 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:51 smithi007 ceph-mon[24826]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:51.282 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:51 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:51.282 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:51 smithi007 ceph-mon[24826]: Deploying daemon node-exporter.b on smithi033 2024-04-15T21:48:51.283 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:51 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3906698826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T21:48:51.283 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:51 smithi007 ceph-mon[29541]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:51.283 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:51 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:51.283 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:51 smithi007 ceph-mon[29541]: Deploying daemon node-exporter.b on smithi033 2024-04-15T21:48:51.283 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:51 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3906698826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T21:48:51.307 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-15T21:48:51.307 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd dump --format=json 2024-04-15T21:48:51.425 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:51 smithi033 ceph-mon[25651]: pgmap v14: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:51.425 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:51 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:51.425 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:51 smithi033 ceph-mon[25651]: Deploying daemon node-exporter.b on smithi033 2024-04-15T21:48:51.425 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:51 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3906698826' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T21:48:51.529 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:48:51.772 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:51 smithi033 bash[41021]: Getting image source signatures 2024-04-15T21:48:51.773 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:51 smithi033 bash[41021]: Copying blob sha256:b5db1e299295edf3005515ab7879c1df64a33c185d3a7a23aa4dcaa17d26f7b3 2024-04-15T21:48:51.773 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:51 smithi033 bash[41021]: Copying blob sha256:aa2a8d90b84cb2a9c422e7005cd166a008ccf22ef5d7d4f07128478585ce35ea 2024-04-15T21:48:51.773 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:51 smithi033 bash[41021]: Copying blob sha256:b45d31ee2d7f9f452678a85b0c837c29e12089f31ee8dbac6c8c24dfa4054a30 2024-04-15T21:48:52.031 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:48:51 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:48:51] "GET /metrics HTTP/1.1" 200 191492 "" "Prometheus/2.33.4" 2024-04-15T21:48:53.176 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:52 smithi033 bash[41021]: Copying config sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 2024-04-15T21:48:53.176 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:52 smithi033 bash[41021]: Writing manifest to image destination 2024-04-15T21:48:53.262 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:53 smithi007 ceph-mon[24826]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:53.263 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:53 smithi007 ceph-mon[29541]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:53.486 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:48:53.486 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":60,"fsid":"18ed60a0-fb71-11ee-bc8f-c7b262605968","created":"2024-04-15T21:43:20.122319+0000","modified":"2024-04-15T21:48:26.282785+0000","last_up_change":"2024-04-15T21:48:11.196763+0000","last_in_change":"2024-04-15T21:47:53.879708+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-15T21:46:18.889328+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"57b79fc0-6f79-4eec-9331-1a2f7f45d0f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6802","nonce":2067680297},{"type":"v1","addr":"172.21.15.7:6803","nonce":2067680297}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6804","nonce":2067680297},{"type":"v1","addr":"172.21.15.7:6805","nonce":2067680297}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6808","nonce":2067680297},{"type":"v1","addr":"172.21.15.7:6809","nonce":2067680297}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6806","nonce":2067680297},{"type":"v1","addr":"172.21.15.7:6807","nonce":2067680297}]},"public_addr":"172.21.15.7:6803/2067680297","cluster_addr":"172.21.15.7:6805/2067680297","heartbeat_back_addr":"172.21.15.7:6809/2067680297","heartbeat_front_addr":"172.21.15.7:6807/2067680297","state":["exists","up"]},{"osd":1,"uuid":"e9184457-c172-4ea2-a429-49cd5ce3a01a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6810","nonce":1851194794},{"type":"v1","addr":"172.21.15.7:6811","nonce":1851194794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6812","nonce":1851194794},{"type":"v1","addr":"172.21.15.7:6813","nonce":1851194794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6816","nonce":1851194794},{"type":"v1","addr":"172.21.15.7:6817","nonce":1851194794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6814","nonce":1851194794},{"type":"v1","addr":"172.21.15.7:6815","nonce":1851194794}]},"public_addr":"172.21.15.7:6811/1851194794","cluster_addr":"172.21.15.7:6813/1851194794","heartbeat_back_addr":"172.21.15.7:6817/1851194794","heartbeat_front_addr":"172.21.15.7:6815/1851194794","state":["exists","up"]},{"osd":2,"uuid":"668b55eb-c95a-46d6-8e89-718b5d394d89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6818","nonce":3569809948},{"type":"v1","addr":"172.21.15.7:6819","nonce":3569809948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6820","nonce":3569809948},{"type":"v1","addr":"172.21.15.7:6821","nonce":3569809948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6824","nonce":3569809948},{"type":"v1","addr":"172.21.15.7:6825","nonce":3569809948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6822","nonce":3569809948},{"type":"v1","addr":"172.21.15.7:6823","nonce":3569809948}]},"public_addr":"172.21.15.7:6819/3569809948","cluster_addr":"172.21.15.7:6821/3569809948","heartbeat_back_addr":"172.21.15.7:6825/3569809948","heartbeat_front_addr":"172.21.15.7:6823/3569809948","state":["exists","up"]},{"osd":3,"uuid":"26df7960-b04a-41c8-8f00-29c0bd10806f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6826","nonce":2388023749},{"type":"v1","addr":"172.21.15.7:6827","nonce":2388023749}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6828","nonce":2388023749},{"type":"v1","addr":"172.21.15.7:6829","nonce":2388023749}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6832","nonce":2388023749},{"type":"v1","addr":"172.21.15.7:6833","nonce":2388023749}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.7:6830","nonce":2388023749},{"type":"v1","addr":"172.21.15.7:6831","nonce":2388023749}]},"public_addr":"172.21.15.7:6827/2388023749","cluster_addr":"172.21.15.7:6829/2388023749","heartbeat_back_addr":"172.21.15.7:6833/2388023749","heartbeat_front_addr":"172.21.15.7:6831/2388023749","state":["exists","up"]},{"osd":4,"uuid":"a3c02631-9af8-4210-81dc-78b3e09d8248","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6800","nonce":993063057},{"type":"v1","addr":"172.21.15.33:6801","nonce":993063057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6802","nonce":993063057},{"type":"v1","addr":"172.21.15.33:6803","nonce":993063057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6806","nonce":993063057},{"type":"v1","addr":"172.21.15.33:6807","nonce":993063057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6804","nonce":993063057},{"type":"v1","addr":"172.21.15.33:6805","nonce":993063057}]},"public_addr":"172.21.15.33:6801/993063057","cluster_addr":"172.21.15.33:6803/993063057","heartbeat_back_addr":"172.21.15.33:6807/993063057","heartbeat_front_addr":"172.21.15.33:6805/993063057","state":["exists","up"]},{"osd":5,"uuid":"ad528133-1de4-4e6d-af65-fd06b811054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6808","nonce":4091940290},{"type":"v1","addr":"172.21.15.33:6809","nonce":4091940290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6810","nonce":4091940290},{"type":"v1","addr":"172.21.15.33:6811","nonce":4091940290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6814","nonce":4091940290},{"type":"v1","addr":"172.21.15.33:6815","nonce":4091940290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6812","nonce":4091940290},{"type":"v1","addr":"172.21.15.33:6813","nonce":4091940290}]},"public_addr":"172.21.15.33:6809/4091940290","cluster_addr":"172.21.15.33:6811/4091940290","heartbeat_back_addr":"172.21.15.33:6815/4091940290","heartbeat_front_addr":"172.21.15.33:6813/4091940290","state":["exists","up"]},{"osd":6,"uuid":"0e9ca75f-ce5b-4fd8-ba69-c5c4175b1d2a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6816","nonce":2157169070},{"type":"v1","addr":"172.21.15.33:6817","nonce":2157169070}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6818","nonce":2157169070},{"type":"v1","addr":"172.21.15.33:6819","nonce":2157169070}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6822","nonce":2157169070},{"type":"v1","addr":"172.21.15.33:6823","nonce":2157169070}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6820","nonce":2157169070},{"type":"v1","addr":"172.21.15.33:6821","nonce":2157169070}]},"public_addr":"172.21.15.33:6817/2157169070","cluster_addr":"172.21.15.33:6819/2157169070","heartbeat_back_addr":"172.21.15.33:6823/2157169070","heartbeat_front_addr":"172.21.15.33:6821/2157169070","state":["exists","up"]},{"osd":7,"uuid":"2b9473c1-b160-4b4d-b86f-0d678e76e9ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6824","nonce":3140679968},{"type":"v1","addr":"172.21.15.33:6825","nonce":3140679968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6826","nonce":3140679968},{"type":"v1","addr":"172.21.15.33:6827","nonce":3140679968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6830","nonce":3140679968},{"type":"v1","addr":"172.21.15.33:6831","nonce":3140679968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.33:6828","nonce":3140679968},{"type":"v1","addr":"172.21.15.33:6829","nonce":3140679968}]},"public_addr":"172.21.15.33:6825/3140679968","cluster_addr":"172.21.15.33:6827/3140679968","heartbeat_back_addr":"172.21.15.33:6831/3140679968","heartbeat_front_addr":"172.21.15.33:6829/3140679968","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:45:29.564043+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:45:51.844504+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:46:14.651534+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:46:37.296913+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:46:59.191741+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:47:22.182751+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:47:45.451353+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-15T21:48:09.275228+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.7:0/3332417937":"2024-04-16T21:48:26.282752+0000","172.21.15.7:0/903497089":"2024-04-16T21:48:26.282752+0000","172.21.15.7:6801/1416545093":"2024-04-16T21:48:26.282752+0000","172.21.15.7:0/2927327763":"2024-04-16T21:48:26.282752+0000","172.21.15.7:0/2974958078":"2024-04-16T21:44:12.524953+0000","172.21.15.7:6801/1363484615":"2024-04-16T21:44:12.524953+0000","172.21.15.7:6800/1363484615":"2024-04-16T21:44:12.524953+0000","172.21.15.7:6800/764447257":"2024-04-16T21:43:50.252562+0000","172.21.15.7:6801/764447257":"2024-04-16T21:43:50.252562+0000","172.21.15.7:0/129134569":"2024-04-16T21:43:50.252562+0000","172.21.15.7:0/1756042719":"2024-04-16T21:48:26.282752+0000","172.21.15.7:6800/1416545093":"2024-04-16T21:48:26.282752+0000","172.21.15.7:0/4220829240":"2024-04-16T21:43:50.252562+0000","172.21.15.7:0/2513010551":"2024-04-16T21:44:12.524953+0000","172.21.15.7:0/4079377254":"2024-04-16T21:44:12.524953+0000","172.21.15.7:0/3034024802":"2024-04-16T21:43:50.252562+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-15T21:48:53.514 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-mon[25651]: pgmap v15: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:53.514 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 podman[41021]: 2024-04-15 21:48:53.229036147 +0000 UTC m=+2.267195505 image pull 1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26 quay.io/prometheus/node-exporter:v1.3.1 2024-04-15T21:48:53.514 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 podman[41021]: 2024-04-15 21:48:53.310460494 +0000 UTC m=+2.348619848 container create 80c30792c19be9b42cc513d39ab44a136f4cc420d34191ab7907e592beaff30e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-15T21:48:53.773 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:48:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:48:53.773 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 podman[41021]: 2024-04-15 21:48:53.512960132 +0000 UTC m=+2.551119486 container init 80c30792c19be9b42cc513d39ab44a136f4cc420d34191ab7907e592beaff30e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-15T21:48:53.773 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 podman[41021]: 2024-04-15 21:48:53.516791049 +0000 UTC m=+2.554950405 container start 80c30792c19be9b42cc513d39ab44a136f4cc420d34191ab7907e592beaff30e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.520Z caller=node_exporter.go:182 level=info msg="Starting node_exporter" version="(version=1.3.1, branch=HEAD, revision=a2321e7b940ddcff26873612bccdf7cd4c42b6b6)" 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.520Z caller=node_exporter.go:183 level=info msg="Build context" build_context="(go=go1.17.3, user=root@243aafa5525c, date=20211205-11:09:49)" 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.521Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/) 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.521Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.521Z caller=node_exporter.go:108 level=info msg="Enabled collectors" 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.521Z caller=node_exporter.go:115 level=info collector=arp 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.521Z caller=node_exporter.go:115 level=info collector=bcache 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.521Z caller=node_exporter.go:115 level=info collector=bonding 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.521Z caller=node_exporter.go:115 level=info collector=btrfs 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=conntrack 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=cpu 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=cpufreq 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=diskstats 2024-04-15T21:48:53.774 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=dmi 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=edac 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=entropy 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=fibrechannel 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=filefd 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=filesystem 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=hwmon 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=infiniband 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=ipvs 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=loadavg 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=mdadm 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=meminfo 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=netclass 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=netdev 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=netstat 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=nfs 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=nfsd 2024-04-15T21:48:53.775 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=nvme 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=os 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=powersupplyclass 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=pressure 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=rapl 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=schedstat 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=sockstat 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=softnet 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=stat 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=tapestats 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=textfile 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=thermal_zone 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=time 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=udp_queues 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=uname 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=vmstat 2024-04-15T21:48:53.776 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=xfs 2024-04-15T21:48:53.777 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:115 level=info collector=zfs 2024-04-15T21:48:53.777 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=node_exporter.go:199 level=info msg="Listening on" address=:9100 2024-04-15T21:48:53.777 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b[41070]: ts=2024-04-15T21:48:53.522Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false 2024-04-15T21:48:53.777 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 bash[41021]: 80c30792c19be9b42cc513d39ab44a136f4cc420d34191ab7907e592beaff30e 2024-04-15T21:48:53.777 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:48:53 smithi033 systemd[1]: Started Ceph node-exporter.b for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:48:54.275 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:54 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1913574134' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T21:48:54.275 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:54 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:54.275 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:54 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1913574134' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T21:48:54.276 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:54 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:54.300 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-15T21:48:54.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-15T21:48:54.301 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-15T21:48:54.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-15T21:48:54.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-15T21:48:54.302 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-15T21:48:54.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-15T21:48:54.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-15T21:48:54.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:54 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1913574134' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-15T21:48:54.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:54 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:54.732 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:48:54.732 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:48:54.732 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:48:54.733 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:48:54.733 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:48:54.751 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:48:54.799 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:48:55.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:55 smithi033 ceph-mon[25651]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:55.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:55 smithi033 ceph-mon[25651]: Deploying daemon alertmanager.a on smithi007 2024-04-15T21:48:55.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:55 smithi007 ceph-mon[24826]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:55.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:55 smithi007 ceph-mon[24826]: Deploying daemon alertmanager.a on smithi007 2024-04-15T21:48:55.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:55 smithi007 ceph-mon[29541]: pgmap v16: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:55.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:55 smithi007 ceph-mon[29541]: Deploying daemon alertmanager.a on smithi007 2024-04-15T21:48:55.642 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:48:57.483 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:57 smithi007 ceph-mon[24826]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:57.484 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:57 smithi007 ceph-mon[29541]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:57.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:57 smithi033 ceph-mon[25651]: pgmap v17: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:58.691 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:58 smithi007 ceph-mon[24826]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:58.691 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:48:58 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:58 smithi007 ceph-mon[29541]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:58.691 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:48:58 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:48:58.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:58 smithi033 ceph-mon[25651]: pgmap v18: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:48:58.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:48:58 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:00.707 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:00 smithi007 ceph-mon[24826]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:00.708 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:00 smithi007 ceph-mon[29541]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:00.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:00 smithi033 ceph-mon[25651]: pgmap v19: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:02.199 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:49:01 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:49:01] "GET /metrics HTTP/1.1" 200 191480 "" "Prometheus/2.33.4" 2024-04-15T21:49:02.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:02 smithi007 ceph-mon[24826]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:02.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:02 smithi007 ceph-mon[29541]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:02.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:02 smithi033 ceph-mon[25651]: pgmap v20: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:04.023 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:49:03 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:49:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:49:04.682 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:04 smithi033 ceph-mon[25651]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:04.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:04 smithi007 ceph-mon[24826]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:04.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:04 smithi007 ceph-mon[29541]: pgmap v21: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:06.462 INFO:teuthology.orchestra.run.smithi007.stdout:236223201291 2024-04-15T21:49:06.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-15T21:49:06.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:06 smithi033 ceph-mon[25651]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:06.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:06 smithi007 ceph-mon[24826]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:06.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:06 smithi007 ceph-mon[29541]: pgmap v22: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:08.512 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:08 smithi007 ceph-mon[24826]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:08.513 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:08 smithi007 ceph-mon[29541]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:08.747 INFO:teuthology.orchestra.run.smithi007.stdout:34359738413 2024-04-15T21:49:08.747 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-15T21:49:08.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:08 smithi033 ceph-mon[25651]: pgmap v23: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:10.511 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:10 smithi007 ceph-mon[24826]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:10.512 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:10 smithi007 ceph-mon[29541]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:10.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:10 smithi033 ceph-mon[25651]: pgmap v24: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:11.082 INFO:teuthology.orchestra.run.smithi007.stdout:201863462930 2024-04-15T21:49:11.083 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-15T21:49:11.644 INFO:teuthology.orchestra.run.smithi007.stdout:55834574889 2024-04-15T21:49:11.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-15T21:49:11.963 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:49:11 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:49:11] "GET /metrics HTTP/1.1" 200 191480 "" "Prometheus/2.33.4" 2024-04-15T21:49:12.359 INFO:teuthology.orchestra.run.smithi007.stdout:111669149727 2024-04-15T21:49:12.359 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-15T21:49:12.611 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:12 smithi007 ceph-mon[24826]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:12.611 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:12 smithi007 ceph-mon[29541]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:12.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:12 smithi033 ceph-mon[25651]: pgmap v25: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:13.759 INFO:teuthology.orchestra.run.smithi007.stdout:77309411364 2024-04-15T21:49:13.760 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-15T21:49:14.023 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:49:13 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:49:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:49:14.390 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:49:14.398 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:49:14.400 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:49:14.401 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:49:14.403 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:49:14.697 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:14 smithi007 ceph-mon[24826]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:14.697 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:14 smithi007 ceph-mon[29541]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:14.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:14 smithi033 ceph-mon[25651]: pgmap v26: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:14.934 INFO:teuthology.orchestra.run.smithi007.stdout:167503724567 2024-04-15T21:49:14.934 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-15T21:49:16.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:16 smithi033 ceph-mon[25651]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:16.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:16 smithi007 ceph-mon[24826]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:16.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:16 smithi007 ceph-mon[29541]: pgmap v27: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:17.339 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:49:17.703 INFO:teuthology.orchestra.run.smithi007.stdout:133143986204 2024-04-15T21:49:17.703 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-15T21:49:17.777 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:49:17.945 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:17 smithi007 podman[46556]: 2024-04-15 21:49:17.807724389 +0000 UTC m=+0.028393232 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-15T21:49:18.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:18 smithi007 ceph-mon[24826]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:18.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:18 smithi007 ceph-mon[29541]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:18.542 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:18 smithi007 podman[46556]: 2024-04-15 21:49:18.357067006 +0000 UTC m=+0.577735822 volume create 4d810b743b36baedaf564b06e6f47cb8de7aa9834dc6a45ecc9a5f18b5e97f0b 2024-04-15T21:49:18.543 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:18 smithi007 podman[46556]: 2024-04-15 21:49:18.43241018 +0000 UTC m=+0.653078999 container create 3f66dfff2116a1399b3903b1ccfdb0c80b1e82543f1438a21cfdcf1dbe913946 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T21:49:18.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:18 smithi033 ceph-mon[25651]: pgmap v28: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:20.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:20 smithi007 ceph-mon[24826]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:20.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:20 smithi007 ceph-mon[29541]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:20.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:20 smithi033 ceph-mon[25651]: pgmap v29: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:21.149 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:21 smithi007 podman[46556]: 2024-04-15 21:49:21.100720232 +0000 UTC m=+3.321389049 container init 3f66dfff2116a1399b3903b1ccfdb0c80b1e82543f1438a21cfdcf1dbe913946 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T21:49:21.149 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:21 smithi007 podman[46556]: 2024-04-15 21:49:21.104575424 +0000 UTC m=+3.325244246 container start 3f66dfff2116a1399b3903b1ccfdb0c80b1e82543f1438a21cfdcf1dbe913946 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T21:49:21.149 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[46646]: level=info ts=2024-04-15T21:49:21.114Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-15T21:49:21.149 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[46646]: level=info ts=2024-04-15T21:49:21.114Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-15T21:49:21.149 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[46646]: level=info ts=2024-04-15T21:49:21.115Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.7 port=9094 2024-04-15T21:49:21.149 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[46646]: level=info ts=2024-04-15T21:49:21.117Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-15T21:49:21.149 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[46646]: level=info ts=2024-04-15T21:49:21.147Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-15T21:49:21.149 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[46646]: level=info ts=2024-04-15T21:49:21.147Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-15T21:49:21.542 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[46646]: level=info ts=2024-04-15T21:49:21.150Z caller=main.go:518 msg=Listening address=:9093 2024-04-15T21:49:21.542 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[46646]: level=info ts=2024-04-15T21:49:21.150Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-15T21:49:21.542 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:21 smithi007 bash[46556]: 3f66dfff2116a1399b3903b1ccfdb0c80b1e82543f1438a21cfdcf1dbe913946 2024-04-15T21:49:21.542 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:21 smithi007 systemd[1]: Started Ceph alertmanager.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:49:22.042 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:49:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:49:21] "GET /metrics HTTP/1.1" 200 191467 "" "Prometheus/2.33.4" 2024-04-15T21:49:22.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:22 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:22.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:22 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:49:22.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:22 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:49:22.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:22 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:49:22.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:22 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:22.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:22 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:49:22.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:22 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:49:22.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:22 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:49:22.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:22 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:22.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:22 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:49:22.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:22 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:49:22.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:22 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:49:23.157 INFO:teuthology.orchestra.run.smithi007.stdout:34359738416 2024-04-15T21:49:23.271 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:23 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[46646]: level=info ts=2024-04-15T21:49:23.117Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00015957s 2024-04-15T21:49:23.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:23 smithi033 ceph-mon[25651]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:23.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:23 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:23.275 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:23 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3888613743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T21:49:23.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:23 smithi007 ceph-mon[24826]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:23.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:23 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:23.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:23 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3888613743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T21:49:23.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:23 smithi007 ceph-mon[29541]: pgmap v30: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:23.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:23 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:23.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:23 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3888613743' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-15T21:49:23.808 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738413 got 34359738416 for osd.0 2024-04-15T21:49:23.808 DEBUG:teuthology.parallel:result is None 2024-04-15T21:49:23.870 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:49:23 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:49:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:49:24.711 INFO:teuthology.orchestra.run.smithi007.stdout:55834574891 2024-04-15T21:49:25.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:24 smithi007 ceph-mon[24826]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:25.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:24 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:25.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:24 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/961284135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T21:49:25.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:24 smithi007 ceph-mon[29541]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:25.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:24 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:25.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:24 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/961284135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T21:49:25.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:24 smithi033 ceph-mon[25651]: pgmap v31: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:25.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:24 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:25.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:24 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/961284135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-15T21:49:25.506 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574889 got 55834574891 for osd.1 2024-04-15T21:49:25.506 DEBUG:teuthology.parallel:result is None 2024-04-15T21:49:26.934 INFO:teuthology.orchestra.run.smithi007.stdout:167503724570 2024-04-15T21:49:27.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:26 smithi007 ceph-mon[24826]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:27.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:26 smithi007 ceph-mon[29541]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:27.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:26 smithi033 ceph-mon[25651]: pgmap v32: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:27.448 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:49:27.676 INFO:teuthology.orchestra.run.smithi007.stdout:201863462933 2024-04-15T21:49:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/798195957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T21:49:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:49:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:49:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:49:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:49:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/29584576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T21:49:28.173 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/90637025' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T21:49:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/798195957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T21:49:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:49:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:49:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:49:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:49:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/29584576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T21:49:28.174 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:27 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/90637025' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T21:49:28.195 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724567 got 167503724570 for osd.5 2024-04-15T21:49:28.195 DEBUG:teuthology.parallel:result is None 2024-04-15T21:49:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:27 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/798195957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-15T21:49:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:27 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:49:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:27 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:49:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:27 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:49:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:27 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:49:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:27 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/29584576' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-15T21:49:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:27 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/90637025' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-15T21:49:28.273 INFO:teuthology.orchestra.run.smithi007.stdout:236223201296 2024-04-15T21:49:28.796 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201291 got 236223201296 for osd.7 2024-04-15T21:49:28.796 DEBUG:teuthology.parallel:result is None 2024-04-15T21:49:29.019 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:28 smithi007 ceph-mon[24826]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:29.019 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:28 smithi007 ceph-mon[29541]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:29.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:28 smithi033 ceph-mon[25651]: pgmap v33: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:29.563 INFO:teuthology.orchestra.run.smithi007.stdout:77309411367 2024-04-15T21:49:29.641 INFO:teuthology.orchestra.run.smithi007.stdout:111669149731 2024-04-15T21:49:30.503 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149727 got 111669149731 for osd.3 2024-04-15T21:49:30.503 DEBUG:teuthology.parallel:result is None 2024-04-15T21:49:30.683 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:30 smithi007 ceph-mon[24826]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:30.683 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:30 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/2098318181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T21:49:30.683 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:30 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/2316050888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T21:49:30.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:30 smithi007 ceph-mon[29541]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:30.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:30 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/2098318181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T21:49:30.684 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:30 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/2316050888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T21:49:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:30 smithi033 ceph-mon[25651]: pgmap v34: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:30 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/2098318181' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-15T21:49:30.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:30 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/2316050888' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-15T21:49:31.037 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411364 got 77309411367 for osd.2 2024-04-15T21:49:31.037 DEBUG:teuthology.parallel:result is None 2024-04-15T21:49:31.292 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:49:31 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[46646]: level=info ts=2024-04-15T21:49:31.119Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001641093s 2024-04-15T21:49:31.534 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462930 got 201863462933 for osd.6 2024-04-15T21:49:31.534 DEBUG:teuthology.parallel:result is None 2024-04-15T21:49:32.155 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:49:31 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:49:31] "GET /metrics HTTP/1.1" 200 191452 "" "Prometheus/2.33.4" 2024-04-15T21:49:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:32 smithi033 ceph-mon[25651]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:32.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:32 smithi007 ceph-mon[24826]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:32.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:32 smithi007 ceph-mon[29541]: pgmap v35: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:33.488 INFO:teuthology.orchestra.run.smithi007.stdout:133143986207 2024-04-15T21:49:34.023 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:49:33 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:49:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:49:34.133 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986204 got 133143986207 for osd.4 2024-04-15T21:49:34.133 DEBUG:teuthology.parallel:result is None 2024-04-15T21:49:34.133 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-15T21:49:34.133 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T21:49:34.435 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:34 smithi007 ceph-mon[24826]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:34.435 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:34 smithi007 ceph-mon[29541]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:34.435 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:34 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/4104040250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T21:49:34.724 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:49:34.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:34 smithi033 ceph-mon[25651]: pgmap v36: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:34.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:34 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/4104040250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T21:49:34.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:34 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/4104040250' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-15T21:49:36.450 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:49:36.452 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-15T21:49:36.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:36 smithi033 ceph-mon[25651]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:36.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:36 smithi007 ceph-mon[24826]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:36.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:36 smithi007 ceph-mon[29541]: pgmap v37: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:37.251 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-04-15T21:49:35.302129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48964,"kb_used_data":6148,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912252,"statfs":{"total":767960285184,"available":767910146048,"internally_reserved":0,"allocated":6295552,"data_stored":3510623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002190"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-15T21:48:27.384357+0000","last_change":"2024-04-15T21:48:15.210703+0000","last_active":"2024-04-15T21:48:27.384357+0000","last_peered":"2024-04-15T21:48:27.384357+0000","last_clean":"2024-04-15T21:48:27.384357+0000","last_became_active":"2024-04-15T21:48:15.210258+0000","last_became_peered":"2024-04-15T21:48:15.210258+0000","last_unstale":"2024-04-15T21:48:27.384357+0000","last_undegraded":"2024-04-15T21:48:27.384357+0000","last_fullsized":"2024-04-15T21:48:27.384357+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T21:46:19.486687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T21:46:19.486687+0000","last_clean_scrub_stamp":"2024-04-15T21:46:19.486687+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T07:27:27.864021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201298,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6092,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739060,"statfs":{"total":95995035648,"available":95988797440,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.55000000000000004}]},{"osd":1,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.433},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.37}]},{"osd":3,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.5},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.40000000000000002}]},{"osd":4,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.61199999999999999}]},{"osd":5,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.63100000000000001}]},{"osd":6,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.60099999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462935,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.61699999999999999}]},{"osd":1,"last update":"Mon Apr 15 21:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.70999999999999996}]},{"osd":2,"last update":"Mon Apr 15 21:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.67200000000000004}]},{"osd":3,"last update":"Mon Apr 15 21:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.63900000000000001}]},{"osd":4,"last update":"Mon Apr 15 21:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.57399999999999995}]},{"osd":5,"last update":"Mon Apr 15 21:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.48199999999999998}]},{"osd":7,"last update":"Mon Apr 15 21:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.55000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.34100000000000003,"15min":0.318},"min":{"1min":0.252,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.36099999999999999,"15min":0.33800000000000002},"min":{"1min":0.29399999999999998,"5min":0.221,"15min":0.221},"max":{"1min":0.57399999999999995,"5min":0.621,"15min":0.621},"last":0.31}]},{"osd":2,"last update":"Mon Apr 15 21:49:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58699999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41699999999999998,"15min":0.41499999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.67400000000000004,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.35599999999999998}]},{"osd":3,"last update":"Mon Apr 15 21:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45700000000000002,"15min":0.45800000000000002},"min":{"1min":0.35899999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48099999999999998,"15min":0.48199999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.40600000000000003}]},{"osd":4,"last update":"Mon Apr 15 21:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55300000000000005,"15min":0.54700000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.627,"15min":0.58099999999999996},"min":{"1min":0.439,"5min":0.377,"15min":0.377},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":0.54900000000000004}]},{"osd":5,"last update":"Mon Apr 15 21:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.82699999999999996,"15min":0.86299999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84599999999999997,"5min":6.4870000000000001,"15min":6.4870000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.44400000000000001}]},{"osd":6,"last update":"Mon Apr 15 21:48:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":6.5209999999999999,"5min":6.5209999999999999,"15min":6.5209999999999999},"last":0.48699999999999999}]},{"osd":7,"last update":"Mon Apr 15 21:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.68600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738419,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 15 21:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.32300000000000001},"min":{"1min":0.24299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.59599999999999997,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35799999999999998,"15min":0.33800000000000002},"min":{"1min":0.25900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.5,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.44900000000000001}]},{"osd":2,"last update":"Mon Apr 15 21:49:24 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39300000000000002,"15min":0.39200000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.505,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.5},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39200000000000002,"15min":0.38300000000000001},"min":{"1min":0.26800000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.40400000000000003}]},{"osd":3,"last update":"Mon Apr 15 21:48:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.439,"5min":0.435,"15min":0.434},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Apr 15 21:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.59199999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.71099999999999997,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.60899999999999999,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.65600000000000003,"5min":1.125,"15min":1.125},"last":0.622}]},{"osd":5,"last update":"Mon Apr 15 21:49:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60199999999999998,"15min":0.61199999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78400000000000003,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.626,"15min":0.63800000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72599999999999998,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.64400000000000002}]},{"osd":6,"last update":"Mon Apr 15 21:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.68000000000000005}]},{"osd":7,"last update":"Mon Apr 15 21:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.67000000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.378},"min":{"1min":0.26100000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.40600000000000003,"15min":0.38900000000000001},"min":{"1min":0.28999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.42999999999999999}]},{"osd":1,"last update":"Mon Apr 15 21:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.442,"15min":0.432},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.47499999999999998}]},{"osd":3,"last update":"Mon Apr 15 21:48:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45000000000000001,"15min":0.44700000000000001},"min":{"1min":0.35399999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47199999999999998,"15min":0.46999999999999997},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.51100000000000001}]},{"osd":4,"last update":"Mon Apr 15 21:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59799999999999998,"15min":0.59299999999999997},"min":{"1min":0.495,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.60799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41999999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.383,"5min":1.383,"15min":1.383},"last":0.55100000000000005}]},{"osd":5,"last update":"Mon Apr 15 21:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.38700000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.78500000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67700000000000005,"15min":0.68100000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.60099999999999998}]},{"osd":6,"last update":"Mon Apr 15 21:48:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.625}]},{"osd":7,"last update":"Mon Apr 15 21:49:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.67000000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:48:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40500000000000003,"15min":0.40100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.434,"15min":0.438},"min":{"1min":0.32000000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55600000000000005,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.308}]},{"osd":1,"last update":"Mon Apr 15 21:48:42 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.44700000000000001,"15min":0.45200000000000001},"min":{"1min":0.30299999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54700000000000004,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.38800000000000001}]},{"osd":2,"last update":"Mon Apr 15 21:48:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.317,"5min":0.313,"15min":0.313},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.37},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.29399999999999998}]},{"osd":4,"last update":"Mon Apr 15 21:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.70499999999999996,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.68799999999999994}]},{"osd":5,"last update":"Mon Apr 15 21:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.45800000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60299999999999998,"15min":0.60199999999999998},"min":{"1min":0.43099999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.65300000000000002}]},{"osd":6,"last update":"Mon Apr 15 21:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Apr 15 21:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.58899999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986208,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.44600000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55900000000000005,"15min":0.56499999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.44500000000000001}]},{"osd":1,"last update":"Mon Apr 15 21:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.46300000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.32600000000000001}]},{"osd":2,"last update":"Mon Apr 15 21:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.438,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.44700000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.66000000000000003,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.53500000000000003}]},{"osd":3,"last update":"Mon Apr 15 21:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.69499999999999995,"5min":0.752,"15min":0.752},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.57899999999999996},"min":{"1min":0.41199999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78300000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.49199999999999999}]},{"osd":5,"last update":"Mon Apr 15 21:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.436,"15min":0.41899999999999998},"min":{"1min":0.311,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.47099999999999997,"15min":0.45400000000000001},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.56599999999999995}]},{"osd":6,"last update":"Mon Apr 15 21:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.59199999999999997}]},{"osd":7,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.61499999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724572,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":240072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:48:29 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.61299999999999999}]},{"osd":1,"last update":"Mon Apr 15 21:48:29 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.65400000000000003}]},{"osd":2,"last update":"Mon Apr 15 21:48:29 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.499},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.63300000000000001}]},{"osd":3,"last update":"Mon Apr 15 21:48:29 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Apr 15 21:48:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.69599999999999995}]},{"osd":6,"last update":"Mon Apr 15 21:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.73899999999999999}]},{"osd":7,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.79400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T21:49:37.253 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph pg dump --format=json 2024-04-15T21:49:37.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:37 smithi007 ceph-mon[24826]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T21:49:37.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:37 smithi007 ceph-mon[29541]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T21:49:37.586 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:49:37.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:37 smithi033 ceph-mon[25651]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T21:49:38.851 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[24826]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:38.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:38.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:38.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:38.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:38.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T21:49:38.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[24826]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T21:49:38.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:38.852 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[24826]: Deploying daemon grafana.a on smithi033 2024-04-15T21:49:38.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[29541]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:38.852 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:38.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:38.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:38.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:38.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T21:49:38.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[29541]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T21:49:38.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:38.853 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:38 smithi007 ceph-mon[29541]: Deploying daemon grafana.a on smithi033 2024-04-15T21:49:39.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:38 smithi033 ceph-mon[25651]: pgmap v38: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:39.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:38 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:39.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:38 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:39.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:38 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:39.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:38 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:39.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:38 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T21:49:39.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:38 smithi033 ceph-mon[25651]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-15T21:49:39.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:38 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:39.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:38 smithi033 ceph-mon[25651]: Deploying daemon grafana.a on smithi033 2024-04-15T21:49:39.372 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:49:39.374 INFO:teuthology.orchestra.run.smithi007.stderr:dumped all 2024-04-15T21:49:39.940 INFO:teuthology.orchestra.run.smithi007.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-04-15T21:49:39.302820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48964,"kb_used_data":6148,"kb_used_omap":0,"kb_used_meta":42752,"kb_avail":749912252,"statfs":{"total":767960285184,"available":767910146048,"internally_reserved":0,"allocated":6295552,"data_stored":3510623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43778048},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002208"},"pg_stats":[{"pgid":"1.0","version":"60'128","reported_seq":267,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-04-15T21:48:27.384357+0000","last_change":"2024-04-15T21:48:15.210703+0000","last_active":"2024-04-15T21:48:27.384357+0000","last_peered":"2024-04-15T21:48:27.384357+0000","last_clean":"2024-04-15T21:48:27.384357+0000","last_became_active":"2024-04-15T21:48:15.210258+0000","last_became_peered":"2024-04-15T21:48:15.210258+0000","last_unstale":"2024-04-15T21:48:27.384357+0000","last_undegraded":"2024-04-15T21:48:27.384357+0000","last_fullsized":"2024-04-15T21:48:27.384357+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-15T21:46:19.486687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-15T21:46:19.486687+0000","last_clean_scrub_stamp":"2024-04-15T21:46:19.486687+0000","objects_scrubbed":0,"log_size":128,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-17T07:27:27.864021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1597440,"data_stored":1586784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201299,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6092,"kb_used_data":1092,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739060,"statfs":{"total":95995035648,"available":95988797440,"internally_reserved":0,"allocated":1118208,"data_stored":769323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.621}]},{"osd":1,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.68200000000000005}]},{"osd":2,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.59199999999999997}]},{"osd":3,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.63500000000000001}]},{"osd":4,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.42299999999999999}]},{"osd":5,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.46600000000000003}]},{"osd":6,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.501},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.40899999999999997}]}]},{"osd":6,"up_from":47,"seq":201863462936,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6152,"kb_used_data":1088,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739000,"statfs":{"total":95995035648,"available":95988736000,"internally_reserved":0,"allocated":1114112,"data_stored":769000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.39800000000000002}]},{"osd":1,"last update":"Mon Apr 15 21:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.59099999999999997}]},{"osd":2,"last update":"Mon Apr 15 21:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.45400000000000001}]},{"osd":3,"last update":"Mon Apr 15 21:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.63800000000000001}]},{"osd":4,"last update":"Mon Apr 15 21:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.57499999999999996}]},{"osd":5,"last update":"Mon Apr 15 21:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60399999999999998}]},{"osd":7,"last update":"Mon Apr 15 21:49:13 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.623}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6344,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738808,"statfs":{"total":95995035648,"available":95988539392,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:48:58 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.34100000000000003,"15min":0.318},"min":{"1min":0.252,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.36099999999999999,"15min":0.33800000000000002},"min":{"1min":0.29399999999999998,"5min":0.221,"15min":0.221},"max":{"1min":0.57399999999999995,"5min":0.621,"15min":0.621},"last":0.38600000000000001}]},{"osd":2,"last update":"Mon Apr 15 21:49:23 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.40400000000000003,"15min":0.39600000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58699999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.41699999999999998,"15min":0.41499999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.67400000000000004,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.32600000000000001}]},{"osd":3,"last update":"Mon Apr 15 21:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45700000000000002,"15min":0.45800000000000002},"min":{"1min":0.35899999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.48099999999999998,"15min":0.48199999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Apr 15 21:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55300000000000005,"15min":0.54700000000000004},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.90500000000000003,"5min":0.627,"15min":0.58099999999999996},"min":{"1min":0.439,"5min":0.377,"15min":0.377},"max":{"1min":6.4720000000000004,"5min":6.4720000000000004,"15min":6.4720000000000004},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Apr 15 21:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.82699999999999996,"15min":0.86299999999999999},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.84599999999999997,"5min":6.4870000000000001,"15min":6.4870000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.624,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.49199999999999999}]},{"osd":6,"last update":"Mon Apr 15 21:48:49 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":6.5209999999999999,"5min":6.5209999999999999,"15min":6.5209999999999999},"last":0.45900000000000002}]},{"osd":7,"last update":"Mon Apr 15 21:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.57999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738420,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6928,"kb_used_data":1096,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738224,"statfs":{"total":95995035648,"available":95987941376,"internally_reserved":0,"allocated":1122304,"data_stored":769588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Apr 15 21:49:02 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.33800000000000002,"15min":0.32300000000000001},"min":{"1min":0.24299999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.59599999999999997,"5min":0.59799999999999998,"15min":0.59799999999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.35799999999999998,"15min":0.33800000000000002},"min":{"1min":0.25900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.5,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.38500000000000001}]},{"osd":2,"last update":"Mon Apr 15 21:49:24 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39300000000000002,"15min":0.39200000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.505,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.39200000000000002,"15min":0.38300000000000001},"min":{"1min":0.26800000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.37}]},{"osd":3,"last update":"Mon Apr 15 21:48:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.439,"5min":0.435,"15min":0.434},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.33500000000000002}]},{"osd":4,"last update":"Mon Apr 15 21:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.58399999999999996,"15min":0.59199999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.71099999999999997,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.624},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.60899999999999999,"15min":0.621},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.65600000000000003,"5min":1.125,"15min":1.125},"last":0.65700000000000003}]},{"osd":5,"last update":"Mon Apr 15 21:49:27 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.60199999999999998,"15min":0.61199999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.78400000000000003,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.626,"15min":0.63800000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.72599999999999998,"5min":1.2090000000000001,"15min":1.2090000000000001},"last":0.754}]},{"osd":6,"last update":"Mon Apr 15 21:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.60199999999999998}]},{"osd":7,"last update":"Mon Apr 15 21:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.55600000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6280,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738872,"statfs":{"total":95995035648,"available":95988604928,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.378},"min":{"1min":0.26100000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.40600000000000003,"15min":0.38900000000000001},"min":{"1min":0.28999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"last":0.35499999999999998}]},{"osd":1,"last update":"Mon Apr 15 21:49:22 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41699999999999998,"15min":0.40899999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.442,"15min":0.432},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.1879999999999999,"5min":1.1879999999999999,"15min":1.1879999999999999},"last":0.315}]},{"osd":3,"last update":"Mon Apr 15 21:48:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45000000000000001,"15min":0.44700000000000001},"min":{"1min":0.35399999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.47199999999999998,"15min":0.46999999999999997},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.39600000000000002}]},{"osd":4,"last update":"Mon Apr 15 21:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.59799999999999998,"15min":0.59299999999999997},"min":{"1min":0.495,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.624},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.60799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41999999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.383,"5min":1.383,"15min":1.383},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Apr 15 21:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.60499999999999998,"15min":0.60799999999999998},"min":{"1min":0.38700000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.78500000000000003,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.67700000000000005,"15min":0.68100000000000005},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.65500000000000003}]},{"osd":6,"last update":"Mon Apr 15 21:48:49 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.63600000000000001}]},{"osd":7,"last update":"Mon Apr 15 21:49:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.67300000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5704,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739448,"statfs":{"total":95995035648,"available":95989194752,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:48:42 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.40500000000000003,"15min":0.40100000000000002},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.434,"15min":0.438},"min":{"1min":0.32000000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.55600000000000005,"5min":0.57299999999999995,"15min":0.57299999999999995},"last":0.27200000000000002}]},{"osd":1,"last update":"Mon Apr 15 21:48:42 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.44500000000000001,"15min":0.44700000000000001},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.44700000000000001,"15min":0.45200000000000001},"min":{"1min":0.30299999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54700000000000004,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.435}]},{"osd":2,"last update":"Mon Apr 15 21:48:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.317,"5min":0.313,"15min":0.313},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.375},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.28599999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Apr 15 21:49:03 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.57999999999999996,"15min":0.58499999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.70499999999999996,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61899999999999999,"15min":0.62},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.629}]},{"osd":5,"last update":"Mon Apr 15 21:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.60399999999999998,"15min":0.60199999999999998},"min":{"1min":0.45800000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.751},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.60299999999999998,"15min":0.60199999999999998},"min":{"1min":0.43099999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.72599999999999998}]},{"osd":6,"last update":"Mon Apr 15 21:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Apr 15 21:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.58699999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986209,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5768,"kb_used_data":576,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739384,"statfs":{"total":95995035648,"available":95989129216,"internally_reserved":0,"allocated":589824,"data_stored":240660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55700000000000005,"15min":0.55200000000000005},"min":{"1min":0.44600000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.55900000000000005,"15min":0.56499999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.56999999999999995}]},{"osd":1,"last update":"Mon Apr 15 21:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.54900000000000004,"15min":0.54700000000000004},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.46300000000000002,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.44900000000000001}]},{"osd":2,"last update":"Mon Apr 15 21:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.54100000000000004,"15min":0.53900000000000003},"min":{"1min":0.438,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.439},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.44700000000000001,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.66000000000000003,"5min":0.71599999999999997,"15min":0.71599999999999997},"last":0.59799999999999998}]},{"osd":3,"last update":"Mon Apr 15 21:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56699999999999995,"15min":0.57099999999999995},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.69499999999999995,"5min":0.752,"15min":0.752},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.57899999999999996},"min":{"1min":0.41199999999999998,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.78300000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.498}]},{"osd":5,"last update":"Mon Apr 15 21:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.436,"15min":0.41899999999999998},"min":{"1min":0.311,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.47099999999999997,"15min":0.45400000000000001},"min":{"1min":0.37,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.40400000000000003}]},{"osd":6,"last update":"Mon Apr 15 21:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.38200000000000001}]},{"osd":7,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.48499999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724573,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5696,"kb_used_data":568,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739456,"statfs":{"total":95995035648,"available":95989202944,"internally_reserved":0,"allocated":581632,"data_stored":240072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Apr 15 21:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.50900000000000001,"15min":0.502},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.52600000000000002,"15min":0.52200000000000002},"min":{"1min":0.39100000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.76400000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.58399999999999996}]},{"osd":1,"last update":"Mon Apr 15 21:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.505,"15min":0.497},"min":{"1min":0.36499999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.51300000000000001,"15min":0.502},"min":{"1min":0.371,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.88400000000000001}]},{"osd":2,"last update":"Mon Apr 15 21:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.502,"15min":0.495},"min":{"1min":0.34999999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53000000000000003,"15min":0.52100000000000002},"min":{"1min":0.43099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.76100000000000001}]},{"osd":3,"last update":"Mon Apr 15 21:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.497,"15min":0.48299999999999998},"min":{"1min":0.41199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.753,"5min":0.753,"15min":0.753},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.54100000000000004,"15min":0.53500000000000003},"min":{"1min":0.39100000000000001,"5min":0.375,"15min":0.375},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.84099999999999997}]},{"osd":4,"last update":"Mon Apr 15 21:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.41399999999999998,"15min":0.40100000000000002},"min":{"1min":0.32100000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.42899999999999999,"15min":0.41399999999999998},"min":{"1min":0.32700000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.63500000000000001}]},{"osd":6,"last update":"Mon Apr 15 21:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.85499999999999998}]},{"osd":7,"last update":"Mon Apr 15 21:49:14 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.78200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-15T21:49:39.942 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-15T21:49:39.942 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-15T21:49:39.942 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-15T21:49:39.942 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -- ceph health --format=json 2024-04-15T21:49:40.166 INFO:teuthology.orchestra.run.smithi007.stderr:Inferring config /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/mon.a/config 2024-04-15T21:49:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:40 smithi007 ceph-mon[24826]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:40.923 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:40 smithi007 ceph-mon[24826]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T21:49:40.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:40 smithi007 ceph-mon[29541]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:40.923 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:40 smithi007 ceph-mon[29541]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T21:49:41.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:40 smithi033 ceph-mon[25651]: pgmap v39: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:41.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:40 smithi033 ceph-mon[25651]: from='client.24535 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-15T21:49:41.942 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-04-15T21:49:41.942 INFO:teuthology.orchestra.run.smithi007.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-15T21:49:42.239 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:49:41 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:49:41] "GET /metrics HTTP/1.1" 200 191452 "" "Prometheus/2.33.4" 2024-04-15T21:49:42.514 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-15T21:49:42.515 INFO:tasks.cephadm:Setup complete, yielding 2024-04-15T21:49:42.515 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T21:49:42.524 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-04-15T21:49:42.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'radosgw-admin realm create --rgw-realm=r --default' 2024-04-15T21:49:42.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:42 smithi007 ceph-mon[24826]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:42.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:42 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/867983890' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-15T21:49:42.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:42 smithi007 ceph-mon[29541]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:42.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:42 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/867983890' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-15T21:49:43.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:42 smithi033 ceph-mon[25651]: pgmap v40: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:43.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:42 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/867983890' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-15T21:49:44.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:49:43 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:49:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:49:45.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:44 smithi033 ceph-mon[25651]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:45.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:44 smithi007 ceph-mon[24826]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:45.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:44 smithi007 ceph-mon[29541]: pgmap v41: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:45.619 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-15T21:49:45.620 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "58fb3fc5-b0cc-4478-9063-56545758535c", 2024-04-15T21:49:45.620 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "r", 2024-04-15T21:49:45.620 INFO:teuthology.orchestra.run.smithi007.stdout: "current_period": "c260a64b-b5b0-4c42-a5b0-2cc45e0bf62d", 2024-04-15T21:49:45.620 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 1 2024-04-15T21:49:45.620 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-15T21:49:45.913 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:45 smithi007 ceph-mon[24826]: osdmap e61: 8 total, 8 up, 8 in 2024-04-15T21:49:45.913 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:45 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/335765101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-15T21:49:45.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:45 smithi007 ceph-mon[29541]: osdmap e61: 8 total, 8 up, 8 in 2024-04-15T21:49:45.913 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:45 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/335765101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-15T21:49:46.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:45 smithi033 ceph-mon[25651]: osdmap e61: 8 total, 8 up, 8 in 2024-04-15T21:49:46.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:45 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/335765101' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-04-15T21:49:46.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'radosgw-admin zonegroup create --rgw-zonegroup=default --master --default' 2024-04-15T21:49:46.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:46 smithi007 ceph-mon[24826]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:46.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:46 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/335765101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-15T21:49:46.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:46 smithi007 ceph-mon[24826]: osdmap e62: 8 total, 8 up, 8 in 2024-04-15T21:49:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:46 smithi007 ceph-mon[29541]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:46 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/335765101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-15T21:49:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:46 smithi007 ceph-mon[29541]: osdmap e62: 8 total, 8 up, 8 in 2024-04-15T21:49:47.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:46 smithi033 ceph-mon[25651]: pgmap v43: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:47.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:46 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/335765101' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-04-15T21:49:47.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:46 smithi033 ceph-mon[25651]: osdmap e62: 8 total, 8 up, 8 in 2024-04-15T21:49:47.782 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-15T21:49:47.782 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "d97f23cf-7624-4af3-bd21-756c395bdbe3", 2024-04-15T21:49:47.782 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default", 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "default", 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": "true", 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [], 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "", 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [], 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [], 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "", 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "58fb3fc5-b0cc-4478-9063-56545758535c", 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-15T21:49:47.783 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-15T21:49:48.021 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:47 smithi007 ceph-mon[24826]: osdmap e63: 8 total, 8 up, 8 in 2024-04-15T21:49:48.022 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:47 smithi007 ceph-mon[29541]: osdmap e63: 8 total, 8 up, 8 in 2024-04-15T21:49:48.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:47 smithi033 ceph-mon[25651]: osdmap e63: 8 total, 8 up, 8 in 2024-04-15T21:49:48.316 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=z --master --default' 2024-04-15T21:49:48.879 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:48 smithi007 ceph-mon[24826]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:48.880 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:48 smithi007 ceph-mon[29541]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:49.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:48 smithi033 ceph-mon[25651]: pgmap v46: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:49:49.995 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-15T21:49:49.995 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "0e4cb2cd-074d-4fb4-bb31-ebc660fae63d", 2024-04-15T21:49:49.995 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "z", 2024-04-15T21:49:49.995 INFO:teuthology.orchestra.run.smithi007.stdout: "domain_root": "z.rgw.meta:root", 2024-04-15T21:49:49.995 INFO:teuthology.orchestra.run.smithi007.stdout: "control_pool": "z.rgw.control", 2024-04-15T21:49:49.995 INFO:teuthology.orchestra.run.smithi007.stdout: "gc_pool": "z.rgw.log:gc", 2024-04-15T21:49:49.995 INFO:teuthology.orchestra.run.smithi007.stdout: "lc_pool": "z.rgw.log:lc", 2024-04-15T21:49:49.995 INFO:teuthology.orchestra.run.smithi007.stdout: "log_pool": "z.rgw.log", 2024-04-15T21:49:49.995 INFO:teuthology.orchestra.run.smithi007.stdout: "intent_log_pool": "z.rgw.log:intent", 2024-04-15T21:49:49.995 INFO:teuthology.orchestra.run.smithi007.stdout: "usage_log_pool": "z.rgw.log:usage", 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: "roles_pool": "z.rgw.meta:roles", 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: "reshard_pool": "z.rgw.log:reshard", 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: "user_keys_pool": "z.rgw.meta:users.keys", 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: "user_email_pool": "z.rgw.meta:users.email", 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: "user_swift_pool": "z.rgw.meta:users.swift", 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: "user_uid_pool": "z.rgw.meta:users.uid", 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: "otp_pool": "z.rgw.otp", 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: "system_key": { 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: "access_key": "", 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: "secret_key": "" 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_pools": [ 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-15T21:49:49.996 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "default-placement", 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: "val": { 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: "index_pool": "z.rgw.buckets.index", 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": { 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD": { 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: "data_pool": "z.rgw.buckets.data" 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: "data_extra_pool": "z.rgw.buckets.non-ec", 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: "index_type": 0 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "58fb3fc5-b0cc-4478-9063-56545758535c", 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout: "notif_pool": "z.rgw.log:notif" 2024-04-15T21:49:49.997 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-15T21:49:50.698 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'radosgw-admin period update --rgw-realm=r --commit' 2024-04-15T21:49:50.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:50 smithi007 ceph-mon[24826]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-04-15T21:49:50.992 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:50 smithi007 ceph-mon[29541]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-04-15T21:49:51.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:50 smithi033 ceph-mon[25651]: pgmap v47: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 1.8 KiB/s wr, 2 op/s 2024-04-15T21:49:51.891 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:49:51 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:49:51] "GET /metrics HTTP/1.1" 200 191475 "" "Prometheus/2.33.4" 2024-04-15T21:49:53.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:52 smithi033 ceph-mon[25651]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 608 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-15T21:49:53.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:52 smithi007 ceph-mon[24826]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 608 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-15T21:49:53.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:52 smithi007 ceph-mon[29541]: pgmap v48: 33 pgs: 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 608 B/s rd, 1.6 KiB/s wr, 2 op/s 2024-04-15T21:49:53.944 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:53 smithi033 ceph-mon[25651]: osdmap e64: 8 total, 8 up, 8 in 2024-04-15T21:49:53.944 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:53 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1456066007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-15T21:49:53.944 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:53 smithi033 ceph-mon[25651]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-15T21:49:53.944 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:49:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:49:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:49:53.945 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:53 smithi033 systemd[1]: Starting Ceph grafana.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:49:53.945 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:53 smithi033 podman[41756]: 2024-04-15 21:49:53.845138823 +0000 UTC m=+0.017085369 image pull dad864ee21e98e69f4029d1e417aa085001566be0d322fbc75bc6f29b0050c01 quay.io/ceph/ceph-grafana:8.3.5 2024-04-15T21:49:54.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:53 smithi007 ceph-mon[29541]: osdmap e64: 8 total, 8 up, 8 in 2024-04-15T21:49:54.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:53 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1456066007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-15T21:49:54.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:53 smithi007 ceph-mon[29541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-15T21:49:54.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:53 smithi007 ceph-mon[24826]: osdmap e64: 8 total, 8 up, 8 in 2024-04-15T21:49:54.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:53 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1456066007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-15T21:49:54.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:53 smithi007 ceph-mon[24826]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]: dispatch 2024-04-15T21:49:54.197 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:53 smithi033 podman[41756]: 2024-04-15 21:49:53.953433358 +0000 UTC m=+0.125379910 container create ba919d83187eb1802cd94c4660cf4800e8ae16465205a6dc46045da3a7720f80 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public, description=Ceph Grafana Container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, version=8.5, io.buildah.version=1.24.2, name=ubi8, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, build-date=2022-03-28T10:36:18.413762, vcs-type=git, release=236.1648460182, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration) 2024-04-15T21:49:54.197 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 podman[41756]: 2024-04-15 21:49:54.163874947 +0000 UTC m=+0.335821493 container init ba919d83187eb1802cd94c4660cf4800e8ae16465205a6dc46045da3a7720f80 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a, maintainer=Paul Cuzner , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.openshift.tags=base rhel8, name=ubi8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, vendor=Red Hat, Inc., architecture=x86_64, build-date=2022-03-28T10:36:18.413762, vcs-type=git, io.buildah.version=1.24.2, release=236.1648460182, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container) 2024-04-15T21:49:54.197 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 podman[41756]: 2024-04-15 21:49:54.167753504 +0000 UTC m=+0.339700049 container start ba919d83187eb1802cd94c4660cf4800e8ae16465205a6dc46045da3a7720f80 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a, architecture=x86_64, com.redhat.component=ubi8-container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.expose-services=, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, release=236.1648460182, description=Ceph Grafana Container, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, io.buildah.version=1.24.2, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, distribution-scope=public) 2024-04-15T21:49:54.449 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="The state of unified alerting is still not defined. The decision will be made during as we run the database migrations" logger=settings 2024-04-15T21:49:54.449 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=warn msg="falling back to legacy setting of 'min_interval_seconds'; please use the configuration option in the `unified_alerting` section if Grafana 8 alerts are enabled." logger=settings 2024-04-15T21:49:54.449 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2024-04-15T21:49:54.449 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2024-04-15T21:49:54.449 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2024-04-15T21:49:54.449 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2024-04-15T21:49:54.449 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-04-15T21:49:54.450 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-04-15T21:49:54.450 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2024-04-15T21:49:54.450 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2024-04-15T21:49:54.450 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2024-04-15T21:49:54.450 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2024-04-15T21:49:54.450 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2024-04-15T21:49:54.450 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="App mode production" logger=settings 2024-04-15T21:49:54.450 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2024-04-15T21:49:54.450 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=warn msg="SQLite database file has broader permissions than it should" logger=sqlstore path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-04-15T21:49:54.450 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Starting DB migrations" logger=migrator 2024-04-15T21:49:54.450 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2024-04-15T21:49:54.450 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2024-04-15T21:49:54.451 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2024-04-15T21:49:54.451 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2024-04-15T21:49:54.451 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2024-04-15T21:49:54.451 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2024-04-15T21:49:54.451 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2024-04-15T21:49:54.451 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2024-04-15T21:49:54.451 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2024-04-15T21:49:54.451 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2024-04-15T21:49:54.451 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-15T21:49:54.451 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index user.login/user.email" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_service_account column to user" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_email - v1" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_org_id - v1" 2024-04-15T21:49:54.452 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_code - v1" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_temp_user_status - v1" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create temp_user v2" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v2" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v2" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v2" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v2" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy temp_user v1 to v2" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop temp_user_tmp_qwerty" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Set created for temp users that will otherwise prematurely expire" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2024-04-15T21:49:54.453 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2024-04-15T21:49:54.454 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2024-04-15T21:49:54.454 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2024-04-15T21:49:54.454 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_user_id - v1" 2024-04-15T21:49:54.454 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2024-04-15T21:49:54.454 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2024-04-15T21:49:54.454 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2024-04-15T21:49:54.455 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2024-04-15T21:49:54.456 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_title" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="delete tags for deleted dashboards" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="delete stars for deleted dashboards" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_is_folder" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2024-04-15T21:49:54.457 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 bash[41756]: ba919d83187eb1802cd94c4660cf4800e8ae16465205a6dc46045da3a7720f80 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 systemd[1]: Started Ceph grafana.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2024-04-15T21:49:54.458 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add uid column" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid value" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index datasource_org_id_uid" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index datasource_org_id_is_default" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2024-04-15T21:49:54.459 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2024-04-15T21:49:54.460 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2024-04-15T21:49:54.460 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2024-04-15T21:49:54.460 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2024-04-15T21:49:54.460 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2024-04-15T21:49:54.460 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add service account foreign key" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add encrypted dashboard json column" 2024-04-15T21:49:54.461 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2024-04-15T21:49:54.462 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column week_start in preferences" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v2" 2024-04-15T21:49:54.463 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy alert_rule_tag v1 to v2" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop table alert_rule_tag_v1" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-04-15T21:49:54.464 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column secure_settings in alert_notification" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert.settings to mediumtext" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_notification_state_alert_id" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add non-unique index alert_rule_tag_alert_id" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2024-04-15T21:49:54.465 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2024-04-15T21:49:54.466 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2024-04-15T21:49:54.466 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2024-04-15T21:49:54.720 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-mon[25651]: pgmap v50: 65 pgs: 32 unknown, 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 530 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-15T21:49:54.720 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-mon[25651]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-15T21:49:54.720 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-mon[25651]: osdmap e65: 8 total, 8 up, 8 in 2024-04-15T21:49:54.720 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:54.720 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:49:54.721 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:49:54.721 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:49:54.721 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2024-04-15T21:49:54.721 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2024-04-15T21:49:54.721 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2024-04-15T21:49:54.721 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-04-15T21:49:54.721 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-04-15T21:49:54.721 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v3" 2024-04-15T21:49:54.721 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-04-15T21:49:54.721 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy annotation_tag v2 to v3" 2024-04-15T21:49:54.721 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop table annotation_tag_v2" 2024-04-15T21:49:54.722 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2024-04-15T21:49:54.722 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2024-04-15T21:49:54.722 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2024-04-15T21:49:54.722 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2024-04-15T21:49:54.722 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2024-04-15T21:49:54.722 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2024-04-15T21:49:54.722 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2024-04-15T21:49:54.722 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2024-04-15T21:49:54.722 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2024-04-15T21:49:54.722 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2024-04-15T21:49:54.723 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2024-04-15T21:49:54.723 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-04-15T21:49:54.723 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-04-15T21:49:54.723 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2024-04-15T21:49:54.723 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2024-04-15T21:49:54.723 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2024-04-15T21:49:54.723 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2024-04-15T21:49:54.723 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2024-04-15T21:49:54.723 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.team_id" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2024-04-15T21:49:54.724 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_user_id" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_team_id" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_org_id_role" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_permission" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="delete acl rules for deleted dashboards and folders" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-04-15T21:49:54.725 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2024-04-15T21:49:54.726 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_auth_token.user_id" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add revoked_at to the user auth token" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create short_url table v1" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index short_url.org_id-uid" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition table" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition table" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and title columns" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition on org_id and uid columns" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition table data column to mediumtext in mysql" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and title columns" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop index in alert_definition on org_id and uid columns" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and title columns" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index in alert_definition on org_id and uid columns" 2024-04-15T21:49:54.727 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column paused in alert_definition" 2024-04-15T21:49:54.728 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition table" 2024-04-15T21:49:54.728 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="delete alert_definition_version table" 2024-04-15T21:49:54.728 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="recreate alert_definition_version table" 2024-04-15T21:49:54.728 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-04-15T21:49:54.728 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-04-15T21:49:54.728 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_definition_version table data column to mediumtext in mysql" 2024-04-15T21:49:54.729 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_definition_version table" 2024-04-15T21:49:54.729 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_instance table" 2024-04-15T21:49:54.729 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-04-15T21:49:54.729 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_instance table on def_org_id, current_state columns" 2024-04-15T21:49:54.729 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column current_state_end to alert_instance" 2024-04-15T21:49:54.729 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-04-15T21:49:54.729 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="remove index def_org_id, current_state on alert_instance" 2024-04-15T21:49:54.729 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_org_id to rule_org_id in alert_instance" 2024-04-15T21:49:54.729 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="rename def_uid to rule_uid in alert_instance" 2024-04-15T21:49:54.729 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-04-15T21:49:54.729 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index rule_org_id, current_state on alert_instance" 2024-04-15T21:49:54.729 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule table" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and title columns" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id and uid columns" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule table data column to mediumtext in mysql" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="remove unique index from alert_rule on org_id, title columns" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add dashboard_uid column to alert_rule" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add panel_id column to alert_rule" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_rule_version table" 2024-04-15T21:49:54.730 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="alter alert_rule_version table data column to mediumtext in mysql" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column for to alert_rule_version" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column annotations to alert_rule_version" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column labels to alert_rule_version" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id=create_alert_configuration_table 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column default in alert_configuration" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column org_id in alert_configuration" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in alert_configuration table on org_id column" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id=create_ngalert_configuration_table 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index in ngalert_configuration on org_id column" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="clear migration entry \"remove unified alerting data\"" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="move dashboard alerts to unified alerting" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element table v1" 2024-04-15T21:49:54.731 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element org_id-folder_id-name-kind" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create library_element_connection table v1" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index library_element org_id_uid" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="clone move dashboard alerts to unified alerting" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create data_keys table" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create kv_store table v1" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index kv_store.org_id-namespace-key" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="update dashboard_uid and panel_id from existing annotations" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create permission table" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index permission.role_id" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_id_action_scope" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create role table" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column display_name" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add column group_name" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index role.org_id" 2024-04-15T21:49:54.732 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role_org_id_name" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index role_org_id_uid" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create team role table" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.org_id" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_role_org_id_team_id_role_id" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_role.team_id" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create user role table" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.org_id" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_role_org_id_user_id_role_id" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index user_role.user_id" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create builtin role table" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.role_id" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.name" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Add column org_id to builtin_role table" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index builtin_role.org_id" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_org_id_role_id_role" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index role_org_id_uid" 2024-04-15T21:49:54.733 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index role.uid" 2024-04-15T21:49:54.734 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="create seed assignment table" 2024-04-15T21:49:54.734 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index builtin_role_role_name" 2024-04-15T21:49:54.734 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="migrations completed" logger=migrator performed=381 skipped=0 duration=491.522612ms 2024-04-15T21:49:54.734 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Created default organization" logger=sqlstore 2024-04-15T21:49:54.734 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Initialising plugins" logger=plugin.manager 2024-04-15T21:49:54.734 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=input 2024-04-15T21:49:54.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[24826]: pgmap v50: 65 pgs: 32 unknown, 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 530 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-15T21:49:54.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[24826]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-15T21:49:54.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[24826]: osdmap e65: 8 total, 8 up, 8 in 2024-04-15T21:49:54.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:54.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:49:54.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:49:54.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:49:54.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[29541]: pgmap v50: 65 pgs: 32 unknown, 33 active+clean; 578 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 530 B/s rd, 1.4 KiB/s wr, 2 op/s 2024-04-15T21:49:54.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[29541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.log","app": "rgw"}]': finished 2024-04-15T21:49:54.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[29541]: osdmap e65: 8 total, 8 up, 8 in 2024-04-15T21:49:54.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:54.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:49:54.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:49:54.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:54 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:49:55.023 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=grafana-piechart-panel 2024-04-15T21:49:55.023 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Plugin registered" logger=plugin.manager pluginId=vonage-status-panel 2024-04-15T21:49:55.023 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="Live Push Gateway initialization" logger=live.push_http 2024-04-15T21:49:55.023 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-04-15T21:49:55.023 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 uid=P43CA22E17D0F9596 2024-04-15T21:49:55.023 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="warming cache for startup" logger=ngalert 2024-04-15T21:49:55.023 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2024-04-15T21:49:55.023 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:49:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:49:54+0000 lvl=info msg="starting MultiOrg Alertmanager" logger=ngalert.multiorg.alertmanager 2024-04-15T21:49:55.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:55 smithi033 ceph-mon[25651]: osdmap e66: 8 total, 8 up, 8 in 2024-04-15T21:49:55.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:55 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1456066007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-15T21:49:55.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:55 smithi033 ceph-mon[25651]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-15T21:49:55.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:55 smithi033 ceph-mon[25651]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-15T21:49:55.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:55 smithi033 ceph-mon[25651]: osdmap e67: 8 total, 8 up, 8 in 2024-04-15T21:49:55.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:55 smithi007 ceph-mon[24826]: osdmap e66: 8 total, 8 up, 8 in 2024-04-15T21:49:55.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:55 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1456066007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-15T21:49:55.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:55 smithi007 ceph-mon[24826]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-15T21:49:55.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:55 smithi007 ceph-mon[24826]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-15T21:49:55.945 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:55 smithi007 ceph-mon[24826]: osdmap e67: 8 total, 8 up, 8 in 2024-04-15T21:49:55.946 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:55 smithi007 ceph-mon[29541]: osdmap e66: 8 total, 8 up, 8 in 2024-04-15T21:49:55.946 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:55 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1456066007' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-15T21:49:55.946 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:55 smithi007 ceph-mon[29541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]: dispatch 2024-04-15T21:49:55.946 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:55 smithi007 ceph-mon[29541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.control","app": "rgw"}]': finished 2024-04-15T21:49:55.946 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:55 smithi007 ceph-mon[29541]: osdmap e67: 8 total, 8 up, 8 in 2024-04-15T21:49:56.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:56 smithi007 ceph-mon[24826]: pgmap v53: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-15T21:49:56.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:56 smithi007 ceph-mon[24826]: osdmap e68: 8 total, 8 up, 8 in 2024-04-15T21:49:56.986 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:56 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-15T21:49:56.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:56 smithi007 ceph-mon[29541]: pgmap v53: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-15T21:49:56.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:56 smithi007 ceph-mon[29541]: osdmap e68: 8 total, 8 up, 8 in 2024-04-15T21:49:56.987 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:56 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-15T21:49:57.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:56 smithi033 ceph-mon[25651]: pgmap v53: 97 pgs: 32 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 853 B/s wr, 5 op/s 2024-04-15T21:49:57.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:56 smithi033 ceph-mon[25651]: osdmap e68: 8 total, 8 up, 8 in 2024-04-15T21:49:57.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:56 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]: dispatch 2024-04-15T21:49:58.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:57 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:58.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:57 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:58.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:57 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-15T21:49:58.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:57 smithi033 ceph-mon[25651]: osdmap e69: 8 total, 8 up, 8 in 2024-04-15T21:49:58.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:57 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-15T21:49:58.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:57 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:58.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:57 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:58.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:57 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:58.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:57 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-15T21:49:58.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:57 smithi007 ceph-mon[24826]: osdmap e69: 8 total, 8 up, 8 in 2024-04-15T21:49:58.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:57 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-15T21:49:58.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:57 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:58.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:57 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:58.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:57 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:58.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:57 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "z.rgw.meta","app": "rgw"}]': finished 2024-04-15T21:49:58.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:57 smithi007 ceph-mon[29541]: osdmap e69: 8 total, 8 up, 8 in 2024-04-15T21:49:58.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:57 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-04-15T21:49:58.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:57 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:59.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[24826]: pgmap v56: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-15T21:49:59.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:59.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[24826]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-15T21:49:59.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[24826]: Reconfiguring daemon alertmanager.a on smithi007 2024-04-15T21:49:59.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-15T21:49:59.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[24826]: osdmap e70: 8 total, 8 up, 8 in 2024-04-15T21:49:59.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-15T21:49:59.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[29541]: pgmap v56: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-15T21:49:59.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:59.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[29541]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-15T21:49:59.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[29541]: Reconfiguring daemon alertmanager.a on smithi007 2024-04-15T21:49:59.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-15T21:49:59.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[29541]: osdmap e70: 8 total, 8 up, 8 in 2024-04-15T21:49:59.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:49:59 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-15T21:49:59.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:59 smithi033 ceph-mon[25651]: pgmap v56: 129 pgs: 64 unknown, 32 creating+peering, 33 active+clean; 579 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 6.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-04-15T21:49:59.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:59 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:49:59.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:59 smithi033 ceph-mon[25651]: Reconfiguring alertmanager.a (dependencies changed)... 2024-04-15T21:49:59.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:59 smithi033 ceph-mon[25651]: Reconfiguring daemon alertmanager.a on smithi007 2024-04-15T21:49:59.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:59 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-04-15T21:49:59.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:59 smithi033 ceph-mon[25651]: osdmap e70: 8 total, 8 up, 8 in 2024-04-15T21:49:59.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:49:59 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2024-04-15T21:49:59.679 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-15T21:49:59.679 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "73b4553d-73d4-4d1b-95ff-4d0ca614797c", 2024-04-15T21:49:59.679 INFO:teuthology.orchestra.run.smithi007.stdout: "epoch": 1, 2024-04-15T21:49:59.679 INFO:teuthology.orchestra.run.smithi007.stdout: "predecessor_uuid": "c260a64b-b5b0-4c42-a5b0-2cc45e0bf62d", 2024-04-15T21:49:59.679 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_status": [], 2024-04-15T21:49:59.679 INFO:teuthology.orchestra.run.smithi007.stdout: "period_map": { 2024-04-15T21:49:59.679 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "73b4553d-73d4-4d1b-95ff-4d0ca614797c", 2024-04-15T21:49:59.679 INFO:teuthology.orchestra.run.smithi007.stdout: "zonegroups": [ 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "d97f23cf-7624-4af3-bd21-756c395bdbe3", 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default", 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "api_name": "default", 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "is_master": "true", 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [], 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames": [], 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "hostnames_s3website": [], 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "0e4cb2cd-074d-4fb4-bb31-ebc660fae63d", 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "zones": [ 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "id": "0e4cb2cd-074d-4fb4-bb31-ebc660fae63d", 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "z", 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "endpoints": [], 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "log_meta": "false", 2024-04-15T21:49:59.680 INFO:teuthology.orchestra.run.smithi007.stdout: "log_data": "false", 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_index_max_shards": 11, 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: "read_only": "false", 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: "tier_type": "", 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from_all": "true", 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_from": [], 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: "redirect_zone": "" 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: "placement_targets": [ 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: "name": "default-placement", 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: "tags": [], 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: "storage_classes": [ 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: "STANDARD" 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-15T21:49:59.681 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: "default_placement": "default-placement", 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "58fb3fc5-b0cc-4478-9063-56545758535c", 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: "sync_policy": { 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: "groups": [] 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: ], 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: "short_zone_ids": [ 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: { 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: "key": "0e4cb2cd-074d-4fb4-bb31-ebc660fae63d", 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: "val": 2665557833 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-15T21:49:59.682 INFO:teuthology.orchestra.run.smithi007.stdout: ] 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zonegroup": "d97f23cf-7624-4af3-bd21-756c395bdbe3", 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: "master_zone": "0e4cb2cd-074d-4fb4-bb31-ebc660fae63d", 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: "period_config": { 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_quota": { 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: "user_quota": { 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false, 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: "check_on_raw": false, 2024-04-15T21:49:59.683 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size": -1, 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "max_size_kb": 0, 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "max_objects": -1 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "user_ratelimit": { 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "bucket_ratelimit": { 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-04-15T21:49:59.684 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-04-15T21:49:59.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:49:59.685 INFO:teuthology.orchestra.run.smithi007.stdout: "anonymous_ratelimit": { 2024-04-15T21:49:59.685 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_ops": 0, 2024-04-15T21:49:59.685 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_ops": 0, 2024-04-15T21:49:59.685 INFO:teuthology.orchestra.run.smithi007.stdout: "max_read_bytes": 0, 2024-04-15T21:49:59.685 INFO:teuthology.orchestra.run.smithi007.stdout: "max_write_bytes": 0, 2024-04-15T21:49:59.685 INFO:teuthology.orchestra.run.smithi007.stdout: "enabled": false 2024-04-15T21:49:59.685 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-15T21:49:59.685 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:49:59.685 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_id": "58fb3fc5-b0cc-4478-9063-56545758535c", 2024-04-15T21:49:59.685 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_name": "r", 2024-04-15T21:49:59.685 INFO:teuthology.orchestra.run.smithi007.stdout: "realm_epoch": 2 2024-04-15T21:49:59.685 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-15T21:50:00.448 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch apply rgw foo --realm r --zone z --placement=2 --port=8000' 2024-04-15T21:50:00.712 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:00 smithi007 ceph-mon[29541]: pgmap v59: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-15T21:50:00.712 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:00 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-15T21:50:00.712 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:00 smithi007 ceph-mon[29541]: osdmap e71: 8 total, 8 up, 8 in 2024-04-15T21:50:00.712 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:00 smithi007 ceph-mon[29541]: overall HEALTH_OK 2024-04-15T21:50:00.712 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:00 smithi007 ceph-mon[24826]: pgmap v59: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-15T21:50:00.712 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:00 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-15T21:50:00.713 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:00 smithi007 ceph-mon[24826]: osdmap e71: 8 total, 8 up, 8 in 2024-04-15T21:50:00.713 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:00 smithi007 ceph-mon[24826]: overall HEALTH_OK 2024-04-15T21:50:01.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:00 smithi033 ceph-mon[25651]: pgmap v59: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-04-15T21:50:01.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:00 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3435967833' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "z.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2024-04-15T21:50:01.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:00 smithi033 ceph-mon[25651]: osdmap e71: 8 total, 8 up, 8 in 2024-04-15T21:50:01.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:00 smithi033 ceph-mon[25651]: overall HEALTH_OK 2024-04-15T21:50:02.042 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:01 smithi007 systemd[1]: Stopping Ceph alertmanager.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:50:02.042 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:50:01 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:50:01] "GET /metrics HTTP/1.1" 200 196424 "" "Prometheus/2.33.4" 2024-04-15T21:50:02.542 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:02 smithi007 bash[49402]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager.a" found: no such container 2024-04-15T21:50:02.920 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:02 smithi007 ceph-mon[24826]: pgmap v61: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 218 B/s rd, 436 B/s wr, 1 op/s 2024-04-15T21:50:02.920 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:02 smithi007 ceph-mon[29541]: pgmap v61: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 218 B/s rd, 436 B/s wr, 1 op/s 2024-04-15T21:50:02.920 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:02 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[46646]: level=info ts=2024-04-15T21:50:02.634Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-15T21:50:02.920 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:02 smithi007 podman[49418]: 2024-04-15 21:50:02.655285616 +0000 UTC m=+0.576849603 container died 3f66dfff2116a1399b3903b1ccfdb0c80b1e82543f1438a21cfdcf1dbe913946 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T21:50:03.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:02 smithi033 ceph-mon[25651]: pgmap v61: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 218 B/s rd, 436 B/s wr, 1 op/s 2024-04-15T21:50:03.172 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:03 smithi007 podman[49418]: 2024-04-15 21:50:03.080597472 +0000 UTC m=+1.002161459 container cleanup 3f66dfff2116a1399b3903b1ccfdb0c80b1e82543f1438a21cfdcf1dbe913946 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T21:50:03.172 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:03 smithi007 bash[49418]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a 2024-04-15T21:50:03.698 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:03 smithi007 bash[49462]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager.a" found: no such container 2024-04-15T21:50:03.951 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:03 smithi007 podman[49448]: 2024-04-15 21:50:03.874524764 +0000 UTC m=+1.217746556 container remove 3f66dfff2116a1399b3903b1ccfdb0c80b1e82543f1438a21cfdcf1dbe913946 (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T21:50:03.951 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:03 smithi007 podman[49448]: 2024-04-15 21:50:03.950173024 +0000 UTC m=+1.293394819 volume remove 4d810b743b36baedaf564b06e6f47cb8de7aa9834dc6a45ecc9a5f18b5e97f0b 2024-04-15T21:50:04.023 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:50:03 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:50:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:50:04.227 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:03 smithi007 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-15T21:50:04.227 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:03 smithi007 systemd[1]: Stopped Ceph alertmanager.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:50:04.227 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:03 smithi007 systemd[1]: Starting Ceph alertmanager.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:50:04.471 INFO:teuthology.orchestra.run.smithi007.stdout:Scheduled rgw.foo update... 2024-04-15T21:50:04.490 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 podman[49560]: 2024-04-15 21:50:04.264457818 +0000 UTC m=+0.020311802 image pull ba2b418f427c0636d654de8757e830c80168e76482bcc46bb2138e569d6c91d4 quay.io/prometheus/alertmanager:v0.23.0 2024-04-15T21:50:04.490 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 podman[49560]: 2024-04-15 21:50:04.369134894 +0000 UTC m=+0.124988847 volume create 37d00418590f7641e540a5b0ef496ebd73615bfdca191e684fa6a7f11b95d47a 2024-04-15T21:50:04.490 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 podman[49560]: 2024-04-15 21:50:04.444444266 +0000 UTC m=+0.200298211 container create 9aff9a2b17564d335a36252f682389279da2e17d10d40bddd5510f7097cadd5a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T21:50:04.765 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:04 smithi007 ceph-mon[24826]: pgmap v62: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-15T21:50:04.765 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:04 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:04.766 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:04 smithi007 ceph-mon[29541]: pgmap v62: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-15T21:50:04.766 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:04 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:04.940 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:04 smithi033 ceph-mon[25651]: pgmap v62: 129 pgs: 129 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-15T21:50:04.941 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:04 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:05.042 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 podman[49560]: 2024-04-15 21:50:04.76473567 +0000 UTC m=+0.520589621 container init 9aff9a2b17564d335a36252f682389279da2e17d10d40bddd5510f7097cadd5a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T21:50:05.042 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 podman[49560]: 2024-04-15 21:50:04.767785708 +0000 UTC m=+0.523639664 container start 9aff9a2b17564d335a36252f682389279da2e17d10d40bddd5510f7097cadd5a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T21:50:05.042 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=info ts=2024-04-15T21:50:04.777Z caller=main.go:225 msg="Starting Alertmanager" version="(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)" 2024-04-15T21:50:05.042 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=info ts=2024-04-15T21:50:04.777Z caller=main.go:226 build_context="(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)" 2024-04-15T21:50:05.042 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=info ts=2024-04-15T21:50:04.778Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.21.15.7 port=9094 2024-04-15T21:50:05.042 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=info ts=2024-04-15T21:50:04.779Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-04-15T21:50:05.042 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=info ts=2024-04-15T21:50:04.805Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-15T21:50:05.042 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=info ts=2024-04-15T21:50:04.805Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-04-15T21:50:05.043 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=info ts=2024-04-15T21:50:04.808Z caller=main.go:518 msg=Listening address=:9093 2024-04-15T21:50:05.043 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=info ts=2024-04-15T21:50:04.808Z caller=tls_config.go:191 msg="TLS is disabled." http2=false 2024-04-15T21:50:05.043 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 bash[49560]: 9aff9a2b17564d335a36252f682389279da2e17d10d40bddd5510f7097cadd5a 2024-04-15T21:50:05.043 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:04 smithi007 systemd[1]: Started Ceph alertmanager.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:50:05.417 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch apply rgw smpl' 2024-04-15T21:50:06.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:05 smithi007 ceph-mon[24826]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:50:06.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:05 smithi007 ceph-mon[24826]: Saving service rgw.foo spec with placement count:2 2024-04-15T21:50:06.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:05 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:06.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:05 smithi007 ceph-mon[24826]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-15T21:50:06.022 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:05 smithi007 ceph-mon[24826]: Reconfiguring daemon prometheus.a on smithi033 2024-04-15T21:50:06.022 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:05 smithi007 ceph-mon[29541]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:50:06.023 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:05 smithi007 ceph-mon[29541]: Saving service rgw.foo spec with placement count:2 2024-04-15T21:50:06.023 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:05 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:06.023 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:05 smithi007 ceph-mon[29541]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-15T21:50:06.023 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:05 smithi007 ceph-mon[29541]: Reconfiguring daemon prometheus.a on smithi033 2024-04-15T21:50:06.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:05 smithi033 ceph-mon[25651]: from='client.14706 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo", "realm": "r", "zone": "z", "placement": "2", "port": 8000, "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:50:06.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:05 smithi033 ceph-mon[25651]: Saving service rgw.foo spec with placement count:2 2024-04-15T21:50:06.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:05 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:06.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:05 smithi033 ceph-mon[25651]: Reconfiguring prometheus.a (dependencies changed)... 2024-04-15T21:50:06.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:05 smithi033 ceph-mon[25651]: Reconfiguring daemon prometheus.a on smithi033 2024-04-15T21:50:07.019 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:06 smithi033 ceph-mon[25651]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-15T21:50:07.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:06 smithi007 ceph-mon[24826]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-15T21:50:07.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:06 smithi007 ceph-mon[29541]: pgmap v63: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-04-15T21:50:07.042 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:06 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=info ts=2024-04-15T21:50:06.779Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000177973s 2024-04-15T21:50:07.186 INFO:teuthology.orchestra.run.smithi007.stdout:Scheduled rgw.smpl update... 2024-04-15T21:50:07.800 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph osd pool create foo' 2024-04-15T21:50:08.188 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:07 smithi033 systemd[1]: Stopping Ceph prometheus.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:50:08.188 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:07 smithi033 bash[42455]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus.a" found: no such container 2024-04-15T21:50:08.188 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:50:08.017Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-15T21:50:08.188 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:50:08.018Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-15T21:50:08.188 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:50:08.018Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-15T21:50:08.188 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:50:08.018Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-15T21:50:08.188 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:50:08.018Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-15T21:50:08.189 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:50:08.018Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-15T21:50:08.189 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:50:08.018Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-15T21:50:08.189 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:50:08.018Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-15T21:50:08.189 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:50:08.018Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-15T21:50:08.189 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:50:08.018Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-15T21:50:08.189 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:50:08.018Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-15T21:50:08.189 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[40796]: ts=2024-04-15T21:50:08.018Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-15T21:50:08.189 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 podman[42468]: 2024-04-15 21:50:08.028391403 +0000 UTC m=+0.123410928 container died 58d4b7cad8f16082714994e87ae2d80de1b132649ee31a67c0797bb3b6103619 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T21:50:08.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:08 smithi007 ceph-mon[24826]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:50:08.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:08 smithi007 ceph-mon[24826]: Saving service rgw.smpl spec with placement count:2 2024-04-15T21:50:08.444 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:08 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:08.444 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:08 smithi007 ceph-mon[29541]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:50:08.444 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:08 smithi007 ceph-mon[29541]: Saving service rgw.smpl spec with placement count:2 2024-04-15T21:50:08.444 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:08 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:08.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-mon[25651]: from='client.24613 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "smpl", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:50:08.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-mon[25651]: Saving service rgw.smpl spec with placement count:2 2024-04-15T21:50:08.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:08 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:08.523 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 podman[42468]: 2024-04-15 21:50:08.342142052 +0000 UTC m=+0.437161577 container cleanup 58d4b7cad8f16082714994e87ae2d80de1b132649ee31a67c0797bb3b6103619 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T21:50:08.523 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 bash[42468]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a 2024-04-15T21:50:08.523 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 bash[42495]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus.a" found: no such container 2024-04-15T21:50:08.828 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 podman[42483]: 2024-04-15 21:50:08.568096443 +0000 UTC m=+0.537909744 container remove 58d4b7cad8f16082714994e87ae2d80de1b132649ee31a67c0797bb3b6103619 (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T21:50:08.828 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-04-15T21:50:08.828 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 systemd[1]: Stopped Ceph prometheus.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:50:08.829 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 systemd[1]: Starting Ceph prometheus.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:50:08.829 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 podman[42562]: 2024-04-15 21:50:08.714352592 +0000 UTC m=+0.019149613 image pull 514e6a882f6e74806a5856468489eeff8d7106095557578da96935e4d0ba4d9d quay.io/prometheus/prometheus:v2.33.4 2024-04-15T21:50:08.829 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:08 smithi033 podman[42562]: 2024-04-15 21:50:08.827551161 +0000 UTC m=+0.132348204 container create 2102c8d49f6e2bacafafec3d992c6d112b47c294031e74a162fc9aa0c298740e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T21:50:09.083 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 podman[42562]: 2024-04-15 21:50:09.054652801 +0000 UTC m=+0.359449803 container init 2102c8d49f6e2bacafafec3d992c6d112b47c294031e74a162fc9aa0c298740e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T21:50:09.083 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 podman[42562]: 2024-04-15 21:50:09.057285269 +0000 UTC m=+0.362082271 container start 2102c8d49f6e2bacafafec3d992c6d112b47c294031e74a162fc9aa0c298740e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T21:50:09.343 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-mon[25651]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 824 B/s rd, 824 B/s wr, 2 op/s 2024-04-15T21:50:09.343 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:09.343 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T21:50:09.343 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2024-04-15T21:50:09.343 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.081Z caller=main.go:475 level=info msg="No time or size retention was set so using the default time retention" duration=15d 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.082Z caller=main.go:512 level=info msg="Starting Prometheus" version="(version=2.33.4, branch=HEAD, revision=83032011a5d3e6102624fe58241a374a7201fee8)" 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.082Z caller=main.go:517 level=info build_context="(go=go1.17.7, user=root@d13bf69e7be8, date=20220222-16:51:28)" 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.082Z caller=main.go:518 level=info host_details="(Linux 5.14.0-438.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Apr 12 13:55:21 UTC 2024 x86_64 smithi033 (none))" 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.082Z caller=main.go:519 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.082Z caller=main.go:520 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.083Z caller=web.go:570 level=info component=web msg="Start listening for connections" address=:9095 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.084Z caller=main.go:923 level=info msg="Starting TSDB ..." 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.085Z caller=tls_config.go:195 level=info component=web msg="TLS is disabled." http2=false 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.087Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.087Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.527µs 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.087Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 bash[42562]: 2102c8d49f6e2bacafafec3d992c6d112b47c294031e74a162fc9aa0c298740e 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 systemd[1]: Started Ceph prometheus.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.323Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-04-15T21:50:09.344 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.323Z caller=head.go:604 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-04-15T21:50:09.345 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.323Z caller=head.go:610 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.833µs wal_replay_duration=236.115126ms total_replay_duration=236.154972ms 2024-04-15T21:50:09.345 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.325Z caller=main.go:944 level=info fs_type=XFS_SUPER_MAGIC 2024-04-15T21:50:09.345 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.325Z caller=main.go:947 level=info msg="TSDB started" 2024-04-15T21:50:09.345 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.325Z caller=main.go:1128 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-04-15T21:50:09.345 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.342Z caller=main.go:1165 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=16.622962ms db_storage=771ns remote_storage=1.111µs web_handler=311ns query_engine=694ns scrape=361.885µs scrape_sd=30.381µs notify=19.653µs notify_sd=7.247µs rules=15.964214ms 2024-04-15T21:50:09.345 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:50:09 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:50:09.342Z caller=main.go:896 level=info msg="Server is ready to receive web requests." 2024-04-15T21:50:09.478 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:09 smithi007 ceph-mon[24826]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 824 B/s rd, 824 B/s wr, 2 op/s 2024-04-15T21:50:09.478 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:09 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:09.478 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:09 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T21:50:09.478 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:09 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2024-04-15T21:50:09.479 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:09 smithi007 ceph-mon[29541]: pgmap v64: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 824 B/s rd, 824 B/s wr, 2 op/s 2024-04-15T21:50:09.479 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:09 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:09.479 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:09 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T21:50:09.479 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:09 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2024-04-15T21:50:10.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T21:50:10.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.33:9095"}]: dispatch 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.33:9095"}]: dispatch 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.33:3000"}]: dispatch 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.33:3000"}]: dispatch 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:50:10.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1464169294' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-15T21:50:10.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T21:50:10.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2024-04-15T21:50:10.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:10.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T21:50:10.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T21:50:10.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.33:9095"}]: dispatch 2024-04-15T21:50:10.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.33:9095"}]: dispatch 2024-04-15T21:50:10.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:10.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T21:50:10.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T21:50:10.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.33:3000"}]: dispatch 2024-04-15T21:50:10.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.33:3000"}]: dispatch 2024-04-15T21:50:10.294 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:10.295 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:50:10.295 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:10.295 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:50:10.295 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:10 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1464169294' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-15T21:50:10.449 INFO:teuthology.orchestra.run.smithi007.stderr:pool 'foo' created 2024-04-15T21:50:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-15T21:50:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.7:9093"}]: dispatch 2024-04-15T21:50:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T21:50:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-15T21:50:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.33:9095"}]: dispatch 2024-04-15T21:50:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.33:9095"}]: dispatch 2024-04-15T21:50:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:10.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T21:50:10.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-15T21:50:10.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.33:3000"}]: dispatch 2024-04-15T21:50:10.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.33:3000"}]: dispatch 2024-04-15T21:50:10.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:10.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:50:10.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:10.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:50:10.524 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:10 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1464169294' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "foo"}]: dispatch 2024-04-15T21:50:11.200 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'rbd pool init foo' 2024-04-15T21:50:11.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:11 smithi033 ceph-mon[25651]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-15T21:50:11.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:11 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1464169294' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-15T21:50:11.512 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:11 smithi033 ceph-mon[25651]: osdmap e72: 8 total, 8 up, 8 in 2024-04-15T21:50:11.538 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:11 smithi007 ceph-mon[24826]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-15T21:50:11.538 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:11 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1464169294' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-15T21:50:11.538 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:11 smithi007 ceph-mon[24826]: osdmap e72: 8 total, 8 up, 8 in 2024-04-15T21:50:11.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:11 smithi007 ceph-mon[29541]: pgmap v65: 129 pgs: 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-15T21:50:11.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:11 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1464169294' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "foo"}]': finished 2024-04-15T21:50:11.538 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:11 smithi007 ceph-mon[29541]: osdmap e72: 8 total, 8 up, 8 in 2024-04-15T21:50:12.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:12 smithi033 ceph-mon[25651]: pgmap v67: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-15T21:50:12.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:12 smithi033 ceph-mon[25651]: osdmap e73: 8 total, 8 up, 8 in 2024-04-15T21:50:12.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:12 smithi007 ceph-mon[24826]: pgmap v67: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-15T21:50:12.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:12 smithi007 ceph-mon[24826]: osdmap e73: 8 total, 8 up, 8 in 2024-04-15T21:50:12.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:12 smithi007 ceph-mon[29541]: pgmap v67: 161 pgs: 32 unknown, 129 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 716 B/s rd, 716 B/s wr, 1 op/s 2024-04-15T21:50:12.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:12 smithi007 ceph-mon[29541]: osdmap e73: 8 total, 8 up, 8 in 2024-04-15T21:50:13.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:13 smithi007 ceph-mon[24826]: osdmap e74: 8 total, 8 up, 8 in 2024-04-15T21:50:13.667 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:13 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:13.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:13 smithi007 ceph-mon[29541]: osdmap e74: 8 total, 8 up, 8 in 2024-04-15T21:50:13.667 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:13 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:13.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:13 smithi033 ceph-mon[25651]: osdmap e74: 8 total, 8 up, 8 in 2024-04-15T21:50:13.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:13 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:14.730 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:14 smithi007 ceph-mon[24826]: pgmap v70: 161 pgs: 22 unknown, 139 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:50:14.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:14 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3068966028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-15T21:50:14.731 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:14 smithi007 ceph-mon[24826]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-15T21:50:14.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:14 smithi007 ceph-mon[29541]: pgmap v70: 161 pgs: 22 unknown, 139 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:50:14.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:14 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3068966028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-15T21:50:14.731 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:14 smithi007 ceph-mon[29541]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-15T21:50:14.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:14 smithi033 ceph-mon[25651]: pgmap v70: 161 pgs: 22 unknown, 139 active+clean; 581 KiB data, 52 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:50:14.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:14 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3068966028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-15T21:50:14.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:14 smithi033 ceph-mon[25651]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]: dispatch 2024-04-15T21:50:15.042 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:14 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=info ts=2024-04-15T21:50:14.781Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.001357659s 2024-04-15T21:50:15.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-15T21:50:15.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: osdmap e75: 8 total, 8 up, 8 in 2024-04-15T21:50:15.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: Saving service rgw.foo spec with placement count:2 2024-04-15T21:50:15.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.qlizaq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:15.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.qlizaq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:15.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.qlizaq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T21:50:15.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:15.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:15 smithi033 ceph-mon[25651]: Deploying daemon rgw.foo.smithi007.qlizaq on smithi007 2024-04-15T21:50:15.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-15T21:50:15.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: osdmap e75: 8 total, 8 up, 8 in 2024-04-15T21:50:15.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: Saving service rgw.foo spec with placement count:2 2024-04-15T21:50:15.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.qlizaq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:15.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.qlizaq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:15.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.qlizaq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T21:50:15.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:15.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[29541]: Deploying daemon rgw.foo.smithi007.qlizaq on smithi007 2024-04-15T21:50:15.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "foo","app": "rbd"}]': finished 2024-04-15T21:50:15.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: osdmap e75: 8 total, 8 up, 8 in 2024-04-15T21:50:15.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: Saving service rgw.foo spec with placement count:2 2024-04-15T21:50:15.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.qlizaq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:15.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.qlizaq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:15.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi007.qlizaq", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T21:50:15.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:15.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:15.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:15 smithi007 ceph-mon[24826]: Deploying daemon rgw.foo.smithi007.qlizaq on smithi007 2024-04-15T21:50:16.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:16 smithi033 ceph-mon[25651]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:50:16.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:16 smithi033 ceph-mon[25651]: osdmap e76: 8 total, 8 up, 8 in 2024-04-15T21:50:16.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:16 smithi007 ceph-mon[24826]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:50:16.782 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:16 smithi007 ceph-mon[24826]: osdmap e76: 8 total, 8 up, 8 in 2024-04-15T21:50:16.783 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:16 smithi007 ceph-mon[29541]: pgmap v72: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:50:16.783 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:16 smithi007 ceph-mon[29541]: osdmap e76: 8 total, 8 up, 8 in 2024-04-15T21:50:17.082 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch apply iscsi foo u p' 2024-04-15T21:50:17.578 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:17 smithi007 ceph-mon[24826]: osdmap e77: 8 total, 8 up, 8 in 2024-04-15T21:50:17.578 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:17 smithi007 ceph-mon[29541]: osdmap e77: 8 total, 8 up, 8 in 2024-04-15T21:50:17.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:17 smithi033 ceph-mon[25651]: osdmap e77: 8 total, 8 up, 8 in 2024-04-15T21:50:19.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:18 smithi033 ceph-mon[25651]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:50:19.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:18 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:19.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:18 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi033.glebqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:19.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:18 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi033.glebqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:19.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:18 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi033.glebqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T21:50:19.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:18 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:19.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:18 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:19.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:18 smithi033 ceph-mon[25651]: Deploying daemon rgw.foo.smithi033.glebqt on smithi033 2024-04-15T21:50:19.185 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[24826]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:50:19.185 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:19.185 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi033.glebqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:19.185 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi033.glebqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:19.185 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi033.glebqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T21:50:19.185 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:19.185 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:19.185 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[24826]: Deploying daemon rgw.foo.smithi033.glebqt on smithi033 2024-04-15T21:50:19.185 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[29541]: pgmap v75: 161 pgs: 161 active+clean; 581 KiB data, 53 MiB used, 715 GiB / 715 GiB avail 2024-04-15T21:50:19.186 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:19.186 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi033.glebqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:19.186 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi033.glebqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:19.186 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.smithi033.glebqt", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T21:50:19.186 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:19.186 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:19.186 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:18 smithi007 ceph-mon[29541]: Deploying daemon rgw.foo.smithi033.glebqt on smithi033 2024-04-15T21:50:19.585 INFO:teuthology.orchestra.run.smithi007.stdout:Scheduled iscsi.foo update... 2024-04-15T21:50:20.327 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'sleep 120' 2024-04-15T21:50:20.799 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[29541]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.8 KiB/s wr, 187 op/s 2024-04-15T21:50:20.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[29541]: from='client.24652 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:50:20.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[29541]: Saving service iscsi.foo spec with placement count:1 2024-04-15T21:50:20.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:20.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:20.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[29541]: Saving service rgw.smpl spec with placement count:2 2024-04-15T21:50:20.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:20.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.uhnuit", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:20.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.uhnuit", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:20.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.uhnuit", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T21:50:20.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:20.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:20.800 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[29541]: Deploying daemon rgw.smpl.smithi007.uhnuit on smithi007 2024-04-15T21:50:20.800 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[24826]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.8 KiB/s wr, 187 op/s 2024-04-15T21:50:20.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[24826]: from='client.24652 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:50:20.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[24826]: Saving service iscsi.foo spec with placement count:1 2024-04-15T21:50:20.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:20.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:20.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[24826]: Saving service rgw.smpl spec with placement count:2 2024-04-15T21:50:20.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:20.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.uhnuit", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:20.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.uhnuit", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:20.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.uhnuit", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T21:50:20.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:20.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:20.801 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:20 smithi007 ceph-mon[24826]: Deploying daemon rgw.smpl.smithi007.uhnuit on smithi007 2024-04-15T21:50:21.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:20 smithi033 ceph-mon[25651]: pgmap v76: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.8 KiB/s wr, 187 op/s 2024-04-15T21:50:21.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:20 smithi033 ceph-mon[25651]: from='client.24652 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "foo", "api_user": "u", "api_password": "p", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:50:21.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:20 smithi033 ceph-mon[25651]: Saving service iscsi.foo spec with placement count:1 2024-04-15T21:50:21.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:20 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:21.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:20 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:21.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:20 smithi033 ceph-mon[25651]: Saving service rgw.smpl spec with placement count:2 2024-04-15T21:50:21.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:20 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:21.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:20 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.uhnuit", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:21.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:20 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.uhnuit", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:21.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:20 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi007.uhnuit", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T21:50:21.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:20 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:21.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:20 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:21.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:20 smithi033 ceph-mon[25651]: Deploying daemon rgw.smpl.smithi007.uhnuit on smithi007 2024-04-15T21:50:22.013 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:50:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:50:21] "GET /metrics HTTP/1.1" 200 197839 "" "Prometheus/2.33.4" 2024-04-15T21:50:22.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:22 smithi007 ceph-mon[24826]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 5.1 KiB/s wr, 164 op/s 2024-04-15T21:50:22.619 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:22 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:22.619 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:22 smithi007 ceph-mon[29541]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 5.1 KiB/s wr, 164 op/s 2024-04-15T21:50:22.619 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:22 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:23.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:22 smithi033 ceph-mon[25651]: pgmap v77: 161 pgs: 161 active+clean; 584 KiB data, 54 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s rd, 5.1 KiB/s wr, 164 op/s 2024-04-15T21:50:23.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:22 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:23.924 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:50:23 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:50:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:50:25.421 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:25 smithi033 ceph-mon[25651]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 4.5 KiB/s wr, 206 op/s 2024-04-15T21:50:25.421 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:25 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:25.421 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:25 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi033.tenllr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:25.421 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:25 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi033.tenllr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:25.421 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:25 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi033.tenllr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T21:50:25.421 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:25 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:25.421 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:25 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:25.421 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:25 smithi033 ceph-mon[25651]: Deploying daemon rgw.smpl.smithi033.tenllr on smithi033 2024-04-15T21:50:25.421 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:25 smithi033 ceph-mon[25651]: osdmap e78: 8 total, 8 up, 8 in 2024-04-15T21:50:25.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[24826]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 4.5 KiB/s wr, 206 op/s 2024-04-15T21:50:25.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:25.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi033.tenllr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:25.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi033.tenllr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:25.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi033.tenllr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T21:50:25.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:25.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:25.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[24826]: Deploying daemon rgw.smpl.smithi033.tenllr on smithi033 2024-04-15T21:50:25.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[24826]: osdmap e78: 8 total, 8 up, 8 in 2024-04-15T21:50:25.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[29541]: pgmap v78: 161 pgs: 161 active+clean; 584 KiB data, 59 MiB used, 715 GiB / 715 GiB avail; 101 KiB/s rd, 4.5 KiB/s wr, 206 op/s 2024-04-15T21:50:25.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:25.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi033.tenllr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:25.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi033.tenllr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-15T21:50:25.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.smpl.smithi033.tenllr", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-15T21:50:25.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:25.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:25.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[29541]: Deploying daemon rgw.smpl.smithi033.tenllr on smithi033 2024-04-15T21:50:25.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:25 smithi007 ceph-mon[29541]: osdmap e78: 8 total, 8 up, 8 in 2024-04-15T21:50:27.373 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:27 smithi033 ceph-mon[25651]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 4.1 KiB/s wr, 240 op/s 2024-04-15T21:50:27.461 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:27 smithi007 ceph-mon[24826]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 4.1 KiB/s wr, 240 op/s 2024-04-15T21:50:27.461 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:27 smithi007 ceph-mon[29541]: pgmap v80: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 126 KiB/s rd, 4.1 KiB/s wr, 240 op/s 2024-04-15T21:50:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2024-04-15T21:50:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-15T21:50:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2024-04-15T21:50:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-04-15T21:50:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-15T21:50:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-15T21:50:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-04-15T21:50:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-15T21:50:28.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:50:28.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:50:28.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:50:28.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:50:28.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:28.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:50:28.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:28.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:50:28.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2024-04-15T21:50:28.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-15T21:50:28.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2024-04-15T21:50:28.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-04-15T21:50:28.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-15T21:50:28.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-15T21:50:28.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-04-15T21:50:28.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-15T21:50:28.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]: dispatch 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-04-15T21:50:28.293 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]: dispatch 2024-04-15T21:50:28.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-04-15T21:50:28.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:50:28.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:50:28.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:50:28.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:50:28.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:28.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:50:28.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:28.294 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:50:29.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:29 smithi007 ceph-mon[24826]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.6 KiB/s wr, 212 op/s 2024-04-15T21:50:29.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:29 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]': finished 2024-04-15T21:50:29.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:29 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-15T21:50:29.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:29 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-04-15T21:50:29.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:29 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-15T21:50:29.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:29 smithi007 ceph-mon[24826]: osdmap e79: 8 total, 8 up, 8 in 2024-04-15T21:50:29.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:29 smithi007 ceph-mon[29541]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.6 KiB/s wr, 212 op/s 2024-04-15T21:50:29.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:29 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]': finished 2024-04-15T21:50:29.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:29 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-15T21:50:29.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:29 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-04-15T21:50:29.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:29 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-15T21:50:29.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:29 smithi007 ceph-mon[29541]: osdmap e79: 8 total, 8 up, 8 in 2024-04-15T21:50:29.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:29 smithi033 ceph-mon[25651]: pgmap v81: 161 pgs: 161 active+clean; 584 KiB data, 60 MiB used, 715 GiB / 715 GiB avail; 111 KiB/s rd, 3.6 KiB/s wr, 212 op/s 2024-04-15T21:50:29.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:29 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.7", "id": [7, 4]}]': finished 2024-04-15T21:50:29.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:29 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-04-15T21:50:29.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:29 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 5]}]': finished 2024-04-15T21:50:29.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:29 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-04-15T21:50:29.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:29 smithi033 ceph-mon[25651]: osdmap e79: 8 total, 8 up, 8 in 2024-04-15T21:50:30.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:30 smithi033 ceph-mon[25651]: osdmap e80: 8 total, 8 up, 8 in 2024-04-15T21:50:30.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:30 smithi007 ceph-mon[24826]: osdmap e80: 8 total, 8 up, 8 in 2024-04-15T21:50:30.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:30 smithi007 ceph-mon[29541]: osdmap e80: 8 total, 8 up, 8 in 2024-04-15T21:50:31.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:31 smithi033 ceph-mon[25651]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 86 MiB used, 715 GiB / 715 GiB avail; 267 KiB/s rd, 341 B/s wr, 412 op/s 2024-04-15T21:50:31.272 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:31 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:31.273 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:31 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:31.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:31 smithi007 ceph-mon[24826]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 86 MiB used, 715 GiB / 715 GiB avail; 267 KiB/s rd, 341 B/s wr, 412 op/s 2024-04-15T21:50:31.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:31 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:31.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:31 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:31.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:31 smithi007 ceph-mon[29541]: pgmap v84: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 86 MiB used, 715 GiB / 715 GiB avail; 267 KiB/s rd, 341 B/s wr, 412 op/s 2024-04-15T21:50:31.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:31 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:31.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:31 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:32.013 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:50:31 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:50:31] "GET /metrics HTTP/1.1" 200 210585 "" "Prometheus/2.33.4" 2024-04-15T21:50:32.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[24826]: Checking dashboard <-> RGW credentials 2024-04-15T21:50:32.748 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:32.749 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:32.749 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:32.749 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.phmsbn", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-15T21:50:32.749 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.phmsbn", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-15T21:50:32.749 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.phmsbn", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-15T21:50:32.749 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:32.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[29541]: Checking dashboard <-> RGW credentials 2024-04-15T21:50:32.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:32.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:32.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:32.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.phmsbn", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-15T21:50:32.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.phmsbn", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-15T21:50:32.749 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.phmsbn", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-15T21:50:32.750 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:32 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:32 smithi033 ceph-mon[25651]: Checking dashboard <-> RGW credentials 2024-04-15T21:50:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:32 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:32 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:32 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:32 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.phmsbn", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-15T21:50:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:32 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.phmsbn", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-04-15T21:50:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:32 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.foo.smithi007.phmsbn", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-04-15T21:50:32.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:32 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:33.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:33 smithi007 ceph-mon[24826]: Checking pool "foo" exists for service iscsi.foo 2024-04-15T21:50:33.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:33 smithi007 ceph-mon[24826]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 86 MiB used, 715 GiB / 715 GiB avail; 189 KiB/s rd, 297 B/s wr, 291 op/s 2024-04-15T21:50:33.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:33 smithi007 ceph-mon[24826]: Deploying daemon iscsi.foo.smithi007.phmsbn on smithi007 2024-04-15T21:50:33.597 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:33 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:33.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:33 smithi007 ceph-mon[29541]: Checking pool "foo" exists for service iscsi.foo 2024-04-15T21:50:33.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:33 smithi007 ceph-mon[29541]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 86 MiB used, 715 GiB / 715 GiB avail; 189 KiB/s rd, 297 B/s wr, 291 op/s 2024-04-15T21:50:33.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:33 smithi007 ceph-mon[29541]: Deploying daemon iscsi.foo.smithi007.phmsbn on smithi007 2024-04-15T21:50:33.598 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:33 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:33.613 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:33 smithi033 ceph-mon[25651]: Checking pool "foo" exists for service iscsi.foo 2024-04-15T21:50:33.614 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:33 smithi033 ceph-mon[25651]: pgmap v85: 161 pgs: 4 peering, 157 active+clean; 584 KiB data, 86 MiB used, 715 GiB / 715 GiB avail; 189 KiB/s rd, 297 B/s wr, 291 op/s 2024-04-15T21:50:33.614 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:33 smithi033 ceph-mon[25651]: Deploying daemon iscsi.foo.smithi007.phmsbn on smithi007 2024-04-15T21:50:33.614 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:33 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:34.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:50:33 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:50:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:50:34.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:34 smithi033 ceph-mon[25651]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s 2024-04-15T21:50:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:34 smithi033 ceph-mon[25651]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-15T21:50:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:34 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:34 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:50:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:34 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:34 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:50:34.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:34 smithi007 ceph-mon[24826]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s 2024-04-15T21:50:34.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:34 smithi007 ceph-mon[24826]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-15T21:50:34.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:34 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:34.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:34 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:50:34.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:34 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:34.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:34 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:50:34.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:34 smithi007 ceph-mon[29541]: pgmap v86: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s 2024-04-15T21:50:34.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:34 smithi007 ceph-mon[29541]: mgrmap e20: y(active, since 2m), standbys: x 2024-04-15T21:50:34.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:34 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:34.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:34 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:50:34.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:34 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:34.543 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:34 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:50:35.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:35 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3544101030' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-15T21:50:35.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:35 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1483810240' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3332417937"}]: dispatch 2024-04-15T21:50:35.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:35 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3332417937"}]: dispatch 2024-04-15T21:50:35.791 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:35 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3544101030' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-15T21:50:35.791 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:35 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1483810240' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3332417937"}]: dispatch 2024-04-15T21:50:35.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:35 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3332417937"}]: dispatch 2024-04-15T21:50:35.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:35 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3544101030' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-15T21:50:35.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:35 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1483810240' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3332417937"}]: dispatch 2024-04-15T21:50:35.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:35 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3332417937"}]: dispatch 2024-04-15T21:50:36.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:36 smithi033 ceph-mon[25651]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s 2024-04-15T21:50:36.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:36 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3332417937"}]': finished 2024-04-15T21:50:36.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:36 smithi033 ceph-mon[25651]: osdmap e81: 8 total, 8 up, 8 in 2024-04-15T21:50:36.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:36 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1212962709' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/903497089"}]: dispatch 2024-04-15T21:50:36.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:36 smithi007 ceph-mon[24826]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s 2024-04-15T21:50:36.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:36 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3332417937"}]': finished 2024-04-15T21:50:36.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:36 smithi007 ceph-mon[24826]: osdmap e81: 8 total, 8 up, 8 in 2024-04-15T21:50:36.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:36 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1212962709' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/903497089"}]: dispatch 2024-04-15T21:50:36.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:36 smithi007 ceph-mon[29541]: pgmap v87: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 167 KiB/s rd, 511 B/s wr, 256 op/s 2024-04-15T21:50:36.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:36 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3332417937"}]': finished 2024-04-15T21:50:36.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:36 smithi007 ceph-mon[29541]: osdmap e81: 8 total, 8 up, 8 in 2024-04-15T21:50:36.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:36 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1212962709' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/903497089"}]: dispatch 2024-04-15T21:50:37.741 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:37 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1212962709' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/903497089"}]': finished 2024-04-15T21:50:37.741 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:37 smithi033 ceph-mon[25651]: osdmap e82: 8 total, 8 up, 8 in 2024-04-15T21:50:37.741 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:37 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3516578547' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1416545093"}]: dispatch 2024-04-15T21:50:37.741 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:37 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1416545093"}]: dispatch 2024-04-15T21:50:37.742 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:37 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:37.742 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:37 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1212962709' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/903497089"}]': finished 2024-04-15T21:50:37.742 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:37 smithi007 ceph-mon[24826]: osdmap e82: 8 total, 8 up, 8 in 2024-04-15T21:50:37.742 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:37 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3516578547' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1416545093"}]: dispatch 2024-04-15T21:50:37.742 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:37 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1416545093"}]: dispatch 2024-04-15T21:50:37.742 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:37 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:37.742 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:37 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1212962709' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/903497089"}]': finished 2024-04-15T21:50:37.743 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:37 smithi007 ceph-mon[29541]: osdmap e82: 8 total, 8 up, 8 in 2024-04-15T21:50:37.743 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:37 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3516578547' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1416545093"}]: dispatch 2024-04-15T21:50:37.743 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:37 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1416545093"}]: dispatch 2024-04-15T21:50:37.743 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:37 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-15T21:50:38.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1416545093"}]': finished 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: osdmap e83: 8 total, 8 up, 8 in 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.543 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:50:38.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:38.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:50:38.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/636744303' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2927327763"}]: dispatch 2024-04-15T21:50:38.544 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2927327763"}]: dispatch 2024-04-15T21:50:38.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-15T21:50:38.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1416545093"}]': finished 2024-04-15T21:50:38.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: osdmap e83: 8 total, 8 up, 8 in 2024-04-15T21:50:38.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-15T21:50:38.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-15T21:50:38.544 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2024-04-15T21:50:38.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-15T21:50:38.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-15T21:50:38.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2024-04-15T21:50:38.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2024-04-15T21:50:38.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:50:38.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:38.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:50:38.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/636744303' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2927327763"}]: dispatch 2024-04-15T21:50:38.545 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:38 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2927327763"}]: dispatch 2024-04-15T21:50:38.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: pgmap v90: 161 pgs: 161 active+clean; 585 KiB data, 91 MiB used, 715 GiB / 715 GiB avail; 5.2 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-15T21:50:38.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1416545093"}]': finished 2024-04-15T21:50:38.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: osdmap e83: 8 total, 8 up, 8 in 2024-04-15T21:50:38.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-15T21:50:38.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2024-04-15T21:50:38.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: Adding iSCSI gateway http://:@172.21.15.7:5000 to Dashboard 2024-04-15T21:50:38.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-15T21:50:38.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2024-04-15T21:50:38.774 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2024-04-15T21:50:38.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi007"}]: dispatch 2024-04-15T21:50:38.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:38.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:50:38.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:50:38.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:50:38.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/636744303' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2927327763"}]: dispatch 2024-04-15T21:50:38.775 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:38 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2927327763"}]: dispatch 2024-04-15T21:50:39.953 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:39 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2927327763"}]': finished 2024-04-15T21:50:39.953 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:39 smithi033 ceph-mon[25651]: osdmap e84: 8 total, 8 up, 8 in 2024-04-15T21:50:39.953 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:39 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3737057380' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2974958078"}]: dispatch 2024-04-15T21:50:39.953 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:39 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2974958078"}]: dispatch 2024-04-15T21:50:39.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:39 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2927327763"}]': finished 2024-04-15T21:50:39.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:39 smithi007 ceph-mon[24826]: osdmap e84: 8 total, 8 up, 8 in 2024-04-15T21:50:39.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:39 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3737057380' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2974958078"}]: dispatch 2024-04-15T21:50:39.992 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:39 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2974958078"}]: dispatch 2024-04-15T21:50:39.992 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:39 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2927327763"}]': finished 2024-04-15T21:50:39.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:39 smithi007 ceph-mon[29541]: osdmap e84: 8 total, 8 up, 8 in 2024-04-15T21:50:39.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:39 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3737057380' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2974958078"}]: dispatch 2024-04-15T21:50:39.993 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:39 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2974958078"}]: dispatch 2024-04-15T21:50:40.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:40 smithi033 ceph-mon[25651]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-04-15T21:50:40.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:40 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2974958078"}]': finished 2024-04-15T21:50:40.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:40 smithi033 ceph-mon[25651]: osdmap e85: 8 total, 8 up, 8 in 2024-04-15T21:50:40.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:40 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/4064799527' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1363484615"}]: dispatch 2024-04-15T21:50:41.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:40 smithi007 ceph-mon[24826]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-04-15T21:50:41.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:40 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2974958078"}]': finished 2024-04-15T21:50:41.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:40 smithi007 ceph-mon[24826]: osdmap e85: 8 total, 8 up, 8 in 2024-04-15T21:50:41.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:40 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/4064799527' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1363484615"}]: dispatch 2024-04-15T21:50:41.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:40 smithi007 ceph-mon[29541]: pgmap v93: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-04-15T21:50:41.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:40 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2974958078"}]': finished 2024-04-15T21:50:41.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:40 smithi007 ceph-mon[29541]: osdmap e85: 8 total, 8 up, 8 in 2024-04-15T21:50:41.043 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:40 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/4064799527' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1363484615"}]: dispatch 2024-04-15T21:50:41.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:41 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/4064799527' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1363484615"}]': finished 2024-04-15T21:50:41.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:41 smithi007 ceph-mon[24826]: osdmap e86: 8 total, 8 up, 8 in 2024-04-15T21:50:41.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:41 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:41.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:41 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/2940953269' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1363484615"}]: dispatch 2024-04-15T21:50:41.994 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:41 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:41.995 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:50:41 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:50:41] "GET /metrics HTTP/1.1" 200 210585 "" "Prometheus/2.33.4" 2024-04-15T21:50:41.995 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:41 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/4064799527' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1363484615"}]': finished 2024-04-15T21:50:41.995 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:41 smithi007 ceph-mon[29541]: osdmap e86: 8 total, 8 up, 8 in 2024-04-15T21:50:41.995 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:41 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:41.995 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:41 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/2940953269' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1363484615"}]: dispatch 2024-04-15T21:50:41.995 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:41 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:42.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:41 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/4064799527' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/1363484615"}]': finished 2024-04-15T21:50:42.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:41 smithi033 ceph-mon[25651]: osdmap e86: 8 total, 8 up, 8 in 2024-04-15T21:50:42.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:41 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:42.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:41 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/2940953269' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1363484615"}]: dispatch 2024-04-15T21:50:42.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:41 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:43.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:42 smithi033 ceph-mon[25651]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-04-15T21:50:43.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:42 smithi033 ceph-mon[25651]: Checking dashboard <-> RGW credentials 2024-04-15T21:50:43.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:42 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:43.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:42 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/2940953269' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1363484615"}]': finished 2024-04-15T21:50:43.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:42 smithi033 ceph-mon[25651]: osdmap e87: 8 total, 8 up, 8 in 2024-04-15T21:50:43.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:42 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/4161675020' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/764447257"}]: dispatch 2024-04-15T21:50:43.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:42 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/764447257"}]: dispatch 2024-04-15T21:50:43.274 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:42 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:43.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[24826]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-04-15T21:50:43.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[24826]: Checking dashboard <-> RGW credentials 2024-04-15T21:50:43.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:43.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/2940953269' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1363484615"}]': finished 2024-04-15T21:50:43.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[24826]: osdmap e87: 8 total, 8 up, 8 in 2024-04-15T21:50:43.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/4161675020' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/764447257"}]: dispatch 2024-04-15T21:50:43.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/764447257"}]: dispatch 2024-04-15T21:50:43.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:43.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[29541]: pgmap v96: 161 pgs: 161 active+clean; 585 KiB data, 92 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 3 op/s 2024-04-15T21:50:43.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[29541]: Checking dashboard <-> RGW credentials 2024-04-15T21:50:43.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:43.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/2940953269' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1363484615"}]': finished 2024-04-15T21:50:43.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[29541]: osdmap e87: 8 total, 8 up, 8 in 2024-04-15T21:50:43.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/4161675020' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/764447257"}]: dispatch 2024-04-15T21:50:43.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/764447257"}]: dispatch 2024-04-15T21:50:43.293 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:42 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:50:43.792 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:43 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=warn ts=2024-04-15T21:50:43.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.33:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.33 because it doesn't contain any IP SANs" 2024-04-15T21:50:43.792 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:43 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=warn ts=2024-04-15T21:50:43.521Z caller=notify.go:724 component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs" 2024-04-15T21:50:44.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:50:43 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:50:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:50:44.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:44 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/764447257"}]': finished 2024-04-15T21:50:44.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:44 smithi033 ceph-mon[25651]: osdmap e88: 8 total, 8 up, 8 in 2024-04-15T21:50:44.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:44 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/2396341157' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/764447257"}]: dispatch 2024-04-15T21:50:44.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:44 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/764447257"}]': finished 2024-04-15T21:50:44.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:44 smithi007 ceph-mon[24826]: osdmap e88: 8 total, 8 up, 8 in 2024-04-15T21:50:44.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:44 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/2396341157' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/764447257"}]: dispatch 2024-04-15T21:50:44.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:44 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/764447257"}]': finished 2024-04-15T21:50:44.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:44 smithi007 ceph-mon[29541]: osdmap e88: 8 total, 8 up, 8 in 2024-04-15T21:50:44.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:44 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/2396341157' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/764447257"}]: dispatch 2024-04-15T21:50:45.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:45 smithi033 ceph-mon[25651]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-15T21:50:45.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:45 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/2396341157' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/764447257"}]': finished 2024-04-15T21:50:45.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:45 smithi033 ceph-mon[25651]: osdmap e89: 8 total, 8 up, 8 in 2024-04-15T21:50:45.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:45 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/444433269' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/129134569"}]: dispatch 2024-04-15T21:50:45.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:45 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/129134569"}]: dispatch 2024-04-15T21:50:45.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:45 smithi007 ceph-mon[24826]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-15T21:50:45.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:45 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/2396341157' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/764447257"}]': finished 2024-04-15T21:50:45.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:45 smithi007 ceph-mon[24826]: osdmap e89: 8 total, 8 up, 8 in 2024-04-15T21:50:45.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:45 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/444433269' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/129134569"}]: dispatch 2024-04-15T21:50:45.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:45 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/129134569"}]: dispatch 2024-04-15T21:50:45.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:45 smithi007 ceph-mon[29541]: pgmap v99: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-15T21:50:45.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:45 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/2396341157' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6801/764447257"}]': finished 2024-04-15T21:50:45.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:45 smithi007 ceph-mon[29541]: osdmap e89: 8 total, 8 up, 8 in 2024-04-15T21:50:45.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:45 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/444433269' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/129134569"}]: dispatch 2024-04-15T21:50:45.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:45 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/129134569"}]: dispatch 2024-04-15T21:50:46.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:46 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:50:46.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:46 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/129134569"}]': finished 2024-04-15T21:50:46.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:46 smithi033 ceph-mon[25651]: osdmap e90: 8 total, 8 up, 8 in 2024-04-15T21:50:46.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:46 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3362307962' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1756042719"}]: dispatch 2024-04-15T21:50:46.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:46 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:50:46.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:46 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/129134569"}]': finished 2024-04-15T21:50:46.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:46 smithi007 ceph-mon[24826]: osdmap e90: 8 total, 8 up, 8 in 2024-04-15T21:50:46.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:46 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3362307962' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1756042719"}]: dispatch 2024-04-15T21:50:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:46 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:50:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:46 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/129134569"}]': finished 2024-04-15T21:50:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:46 smithi007 ceph-mon[29541]: osdmap e90: 8 total, 8 up, 8 in 2024-04-15T21:50:46.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:46 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3362307962' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1756042719"}]: dispatch 2024-04-15T21:50:47.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:47 smithi033 ceph-mon[25651]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 8 op/s 2024-04-15T21:50:47.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:47 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3362307962' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1756042719"}]': finished 2024-04-15T21:50:47.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:47 smithi033 ceph-mon[25651]: osdmap e91: 8 total, 8 up, 8 in 2024-04-15T21:50:47.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:47 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1940211329' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1416545093"}]: dispatch 2024-04-15T21:50:47.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:47 smithi007 ceph-mon[24826]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 8 op/s 2024-04-15T21:50:47.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:47 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3362307962' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1756042719"}]': finished 2024-04-15T21:50:47.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:47 smithi007 ceph-mon[24826]: osdmap e91: 8 total, 8 up, 8 in 2024-04-15T21:50:47.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:47 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1940211329' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1416545093"}]: dispatch 2024-04-15T21:50:47.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:47 smithi007 ceph-mon[29541]: pgmap v101: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 8.0 KiB/s rd, 8 op/s 2024-04-15T21:50:47.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:47 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3362307962' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/1756042719"}]': finished 2024-04-15T21:50:47.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:47 smithi007 ceph-mon[29541]: osdmap e91: 8 total, 8 up, 8 in 2024-04-15T21:50:47.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:47 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1940211329' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1416545093"}]: dispatch 2024-04-15T21:50:48.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:48 smithi033 ceph-mon[25651]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-15T21:50:48.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:48 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1940211329' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1416545093"}]': finished 2024-04-15T21:50:48.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:48 smithi033 ceph-mon[25651]: osdmap e92: 8 total, 8 up, 8 in 2024-04-15T21:50:48.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:48 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1926210837' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4220829240"}]: dispatch 2024-04-15T21:50:48.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:48 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4220829240"}]: dispatch 2024-04-15T21:50:48.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:48 smithi007 ceph-mon[24826]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-15T21:50:48.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:48 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1940211329' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1416545093"}]': finished 2024-04-15T21:50:48.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:48 smithi007 ceph-mon[24826]: osdmap e92: 8 total, 8 up, 8 in 2024-04-15T21:50:48.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:48 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1926210837' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4220829240"}]: dispatch 2024-04-15T21:50:48.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:48 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4220829240"}]: dispatch 2024-04-15T21:50:48.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:48 smithi007 ceph-mon[29541]: pgmap v104: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 9.2 KiB/s rd, 9 op/s 2024-04-15T21:50:48.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:48 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1940211329' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:6800/1416545093"}]': finished 2024-04-15T21:50:48.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:48 smithi007 ceph-mon[29541]: osdmap e92: 8 total, 8 up, 8 in 2024-04-15T21:50:48.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:48 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1926210837' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4220829240"}]: dispatch 2024-04-15T21:50:48.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:48 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4220829240"}]: dispatch 2024-04-15T21:50:49.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:49 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4220829240"}]': finished 2024-04-15T21:50:49.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:49 smithi033 ceph-mon[25651]: osdmap e93: 8 total, 8 up, 8 in 2024-04-15T21:50:49.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:49 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/594427675' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2513010551"}]: dispatch 2024-04-15T21:50:49.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:49 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2513010551"}]: dispatch 2024-04-15T21:50:49.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:49 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4220829240"}]': finished 2024-04-15T21:50:49.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:49 smithi007 ceph-mon[24826]: osdmap e93: 8 total, 8 up, 8 in 2024-04-15T21:50:49.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:49 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/594427675' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2513010551"}]: dispatch 2024-04-15T21:50:49.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:49 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2513010551"}]: dispatch 2024-04-15T21:50:49.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:49 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4220829240"}]': finished 2024-04-15T21:50:49.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:49 smithi007 ceph-mon[29541]: osdmap e93: 8 total, 8 up, 8 in 2024-04-15T21:50:49.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:49 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/594427675' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2513010551"}]: dispatch 2024-04-15T21:50:49.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:49 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2513010551"}]: dispatch 2024-04-15T21:50:50.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:50 smithi033 ceph-mon[25651]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:50:50.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:50 smithi033 ceph-mon[25651]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2513010551"}]': finished 2024-04-15T21:50:50.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:50 smithi033 ceph-mon[25651]: osdmap e94: 8 total, 8 up, 8 in 2024-04-15T21:50:50.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:50 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3869033175' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4079377254"}]: dispatch 2024-04-15T21:50:50.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:50 smithi007 ceph-mon[24826]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:50:50.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:50 smithi007 ceph-mon[24826]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2513010551"}]': finished 2024-04-15T21:50:50.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:50 smithi007 ceph-mon[24826]: osdmap e94: 8 total, 8 up, 8 in 2024-04-15T21:50:50.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:50 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3869033175' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4079377254"}]: dispatch 2024-04-15T21:50:50.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:50 smithi007 ceph-mon[29541]: pgmap v107: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:50:50.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:50 smithi007 ceph-mon[29541]: from='client.? ' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/2513010551"}]': finished 2024-04-15T21:50:50.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:50 smithi007 ceph-mon[29541]: osdmap e94: 8 total, 8 up, 8 in 2024-04-15T21:50:50.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:50 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3869033175' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4079377254"}]: dispatch 2024-04-15T21:50:51.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:51 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3869033175' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4079377254"}]': finished 2024-04-15T21:50:51.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:51 smithi033 ceph-mon[25651]: osdmap e95: 8 total, 8 up, 8 in 2024-04-15T21:50:51.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:51 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1436113012' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3034024802"}]: dispatch 2024-04-15T21:50:51.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:51 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3869033175' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4079377254"}]': finished 2024-04-15T21:50:51.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:51 smithi007 ceph-mon[24826]: osdmap e95: 8 total, 8 up, 8 in 2024-04-15T21:50:51.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:51 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1436113012' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3034024802"}]: dispatch 2024-04-15T21:50:51.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:51 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3869033175' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/4079377254"}]': finished 2024-04-15T21:50:51.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:51 smithi007 ceph-mon[29541]: osdmap e95: 8 total, 8 up, 8 in 2024-04-15T21:50:51.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:51 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1436113012' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3034024802"}]: dispatch 2024-04-15T21:50:52.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:50:51 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:50:51] "GET /metrics HTTP/1.1" 200 214883 "" "Prometheus/2.33.4" 2024-04-15T21:50:52.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:52 smithi033 ceph-mon[25651]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:50:52.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:52 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1436113012' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3034024802"}]': finished 2024-04-15T21:50:52.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:52 smithi033 ceph-mon[25651]: osdmap e96: 8 total, 8 up, 8 in 2024-04-15T21:50:52.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:52 smithi007 ceph-mon[24826]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:50:52.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:52 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1436113012' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3034024802"}]': finished 2024-04-15T21:50:52.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:52 smithi007 ceph-mon[24826]: osdmap e96: 8 total, 8 up, 8 in 2024-04-15T21:50:52.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:52 smithi007 ceph-mon[29541]: pgmap v110: 161 pgs: 161 active+clean; 585 KiB data, 93 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:50:52.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:52 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1436113012' entity='client.iscsi.foo.smithi007.phmsbn' cmd='[{"prefix": "osd blocklist", "blocklistop": "rm", "addr": "172.21.15.7:0/3034024802"}]': finished 2024-04-15T21:50:52.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:52 smithi007 ceph-mon[29541]: osdmap e96: 8 total, 8 up, 8 in 2024-04-15T21:50:53.792 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:50:53 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=error ts=2024-04-15T21:50:53.506Z caller=dispatch.go:354 component=dispatcher msg="Notify for alerts failed" num_alerts=6 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://172.21.15.7:8443//api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.7 because it doesn't contain any IP SANs; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://172.21.15.33:8443/api/prometheus_receiver\": x509: cannot validate certificate for 172.21.15.33 because it doesn't contain any IP SANs" 2024-04-15T21:50:53.940 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:50:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:50:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:50:54.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:54 smithi033 ceph-mon[25651]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-15T21:50:54.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:54 smithi007 ceph-mon[24826]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-15T21:50:54.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:54 smithi007 ceph-mon[29541]: pgmap v112: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-04-15T21:50:55.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:55 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:50:55.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:55 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:50:55.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:55 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:50:56.773 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:56 smithi033 ceph-mon[25651]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:50:56.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:56 smithi007 ceph-mon[24826]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:50:56.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:56 smithi007 ceph-mon[29541]: pgmap v113: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:50:58.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:50:58 smithi033 ceph-mon[25651]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2024-04-15T21:50:58.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:50:58 smithi007 ceph-mon[24826]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2024-04-15T21:50:58.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:50:58 smithi007 ceph-mon[29541]: pgmap v114: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 651 B/s rd, 0 op/s 2024-04-15T21:51:00.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:00 smithi033 ceph-mon[25651]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-15T21:51:00.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:00 smithi007 ceph-mon[24826]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-15T21:51:00.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:00 smithi007 ceph-mon[29541]: pgmap v115: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-04-15T21:51:02.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:51:01 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:51:01] "GET /metrics HTTP/1.1" 200 214899 "" "Prometheus/2.33.4" 2024-04-15T21:51:02.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:02 smithi033 ceph-mon[25651]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-15T21:51:02.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:02 smithi007 ceph-mon[24826]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-15T21:51:02.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:02 smithi007 ceph-mon[29541]: pgmap v116: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-04-15T21:51:04.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:51:03 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:51:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:51:04.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:04 smithi033 ceph-mon[25651]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-15T21:51:04.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:04 smithi007 ceph-mon[24826]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-15T21:51:04.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:04 smithi007 ceph-mon[29541]: pgmap v117: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-04-15T21:51:05.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:05 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:05.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:05 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:05.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:05 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:06.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:06 smithi033 ceph-mon[25651]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:06.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:06 smithi007 ceph-mon[24826]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:06.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:06 smithi007 ceph-mon[29541]: pgmap v118: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:08.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:08 smithi033 ceph-mon[25651]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:08.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:08 smithi007 ceph-mon[24826]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:08.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:08 smithi007 ceph-mon[29541]: pgmap v119: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:10.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:10 smithi033 ceph-mon[25651]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:10.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:10 smithi007 ceph-mon[24826]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:10.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:10 smithi007 ceph-mon[29541]: pgmap v120: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:12.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:51:11 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:51:11] "GET /metrics HTTP/1.1" 200 214899 "" "Prometheus/2.33.4" 2024-04-15T21:51:12.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:12 smithi033 ceph-mon[25651]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:12.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:12 smithi007 ceph-mon[24826]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:12.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:12 smithi007 ceph-mon[29541]: pgmap v121: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:14.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:51:13 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:51:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:51:14.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:14 smithi033 ceph-mon[25651]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:14.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:14 smithi007 ceph-mon[24826]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:14.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:14 smithi007 ceph-mon[29541]: pgmap v122: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:15.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:15 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:15.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:15 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:15.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:15 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:16.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:16 smithi033 ceph-mon[25651]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:16.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:16 smithi007 ceph-mon[24826]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:16.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:16 smithi007 ceph-mon[29541]: pgmap v123: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:18.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:18 smithi007 ceph-mon[24826]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:18.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:18 smithi007 ceph-mon[29541]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:19.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:18 smithi033 ceph-mon[25651]: pgmap v124: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:20.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:20 smithi007 ceph-mon[24826]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:20.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:20 smithi007 ceph-mon[29541]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:21.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:20 smithi033 ceph-mon[25651]: pgmap v125: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:22.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:51:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:51:21] "GET /metrics HTTP/1.1" 200 214859 "" "Prometheus/2.33.4" 2024-04-15T21:51:22.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:22 smithi007 ceph-mon[24826]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:22.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:22 smithi007 ceph-mon[29541]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:23.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:22 smithi033 ceph-mon[25651]: pgmap v126: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:23.923 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:51:23 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:51:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:51:24.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:24 smithi007 ceph-mon[24826]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:24.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:24 smithi007 ceph-mon[29541]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:25.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:24 smithi033 ceph-mon[25651]: pgmap v127: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:25.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:25 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:25.793 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:25 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:26.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:25 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:26.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:26 smithi007 ceph-mon[24826]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:26.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:26 smithi007 ceph-mon[29541]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:27.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:26 smithi033 ceph-mon[25651]: pgmap v128: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:28.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:27 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:51:28.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:27 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:51:28.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:27 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:51:28.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:27 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:51:28.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:27 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:51:28.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:27 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:51:28.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:27 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:51:28.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:27 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:51:28.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:27 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:51:28.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:27 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:51:28.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:27 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:51:28.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:27 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:51:29.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:28 smithi033 ceph-mon[25651]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:29.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:28 smithi007 ceph-mon[24826]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:29.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:28 smithi007 ceph-mon[29541]: pgmap v129: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:31.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:30 smithi033 ceph-mon[25651]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:31.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:30 smithi007 ceph-mon[24826]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:31.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:30 smithi007 ceph-mon[29541]: pgmap v130: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:32.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:51:31 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:51:31] "GET /metrics HTTP/1.1" 200 214810 "" "Prometheus/2.33.4" 2024-04-15T21:51:33.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:32 smithi033 ceph-mon[25651]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:33.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:32 smithi007 ceph-mon[24826]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:33.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:32 smithi007 ceph-mon[29541]: pgmap v131: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:34.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:51:33 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:51:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:51:34.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:34 smithi007 ceph-mon[24826]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:34.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:34 smithi007 ceph-mon[29541]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:35.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:34 smithi033 ceph-mon[25651]: pgmap v132: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:36.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:35 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:36.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:35 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:36.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:35 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:37.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:36 smithi033 ceph-mon[25651]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:37.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:36 smithi007 ceph-mon[24826]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:37.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:36 smithi007 ceph-mon[29541]: pgmap v133: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:39.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:38 smithi033 ceph-mon[25651]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:39.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:38 smithi007 ceph-mon[24826]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:39.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:38 smithi007 ceph-mon[29541]: pgmap v134: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:41.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:40 smithi033 ceph-mon[25651]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:41.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:40 smithi007 ceph-mon[24826]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:41.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:40 smithi007 ceph-mon[29541]: pgmap v135: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:42.151 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:51:41 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:51:41] "GET /metrics HTTP/1.1" 200 214810 "" "Prometheus/2.33.4" 2024-04-15T21:51:43.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:42 smithi033 ceph-mon[25651]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:43.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:42 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:51:43.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:42 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:51:43.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:42 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:51:43.024 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:42 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:51:43.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:42 smithi007 ceph-mon[24826]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:43.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:42 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:51:43.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:42 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:51:43.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:42 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:51:43.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:42 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:51:43.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:42 smithi007 ceph-mon[29541]: pgmap v136: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:43.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:42 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:51:43.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:42 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:51:43.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:42 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:51:43.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:42 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:51:44.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:51:43 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:51:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:51:44.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:44 smithi007 ceph-mon[24826]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:44.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:44 smithi007 ceph-mon[29541]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:45.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:44 smithi033 ceph-mon[25651]: pgmap v137: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:46.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:45 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:46.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:45 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:46.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:45 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:47.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:46 smithi033 ceph-mon[25651]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:47.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:46 smithi007 ceph-mon[24826]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:47.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:46 smithi007 ceph-mon[29541]: pgmap v138: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:49.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:48 smithi033 ceph-mon[25651]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:49.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:48 smithi007 ceph-mon[24826]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:49.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:48 smithi007 ceph-mon[29541]: pgmap v139: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:50.844 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:50 smithi007 ceph-mon[24826]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:50.844 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:50 smithi007 ceph-mon[29541]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:51.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:50 smithi033 ceph-mon[25651]: pgmap v140: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:52.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:51:51 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:51:51] "GET /metrics HTTP/1.1" 200 214807 "" "Prometheus/2.33.4" 2024-04-15T21:51:53.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:52 smithi033 ceph-mon[25651]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:53.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:52 smithi007 ceph-mon[24826]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:53.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:52 smithi007 ceph-mon[29541]: pgmap v141: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:53.924 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:51:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:51:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:51:54.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:54 smithi007 ceph-mon[24826]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:54.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:54 smithi007 ceph-mon[29541]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:54.940 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:54 smithi033 ceph-mon[25651]: pgmap v142: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:51:55.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:55 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:56.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:55 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:56.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:55 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:51:57.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:56 smithi033 ceph-mon[25651]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:57.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:56 smithi007 ceph-mon[24826]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:57.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:56 smithi007 ceph-mon[29541]: pgmap v143: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:59.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:51:58 smithi033 ceph-mon[25651]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:59.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:51:58 smithi007 ceph-mon[24826]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:51:59.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:51:58 smithi007 ceph-mon[29541]: pgmap v144: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:01.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:00 smithi033 ceph-mon[25651]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:01.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:00 smithi007 ceph-mon[24826]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:01.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:00 smithi007 ceph-mon[29541]: pgmap v145: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:02.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:52:01 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:52:01] "GET /metrics HTTP/1.1" 200 214813 "" "Prometheus/2.33.4" 2024-04-15T21:52:03.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:02 smithi033 ceph-mon[25651]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:03.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:02 smithi007 ceph-mon[24826]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:03.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:02 smithi007 ceph-mon[29541]: pgmap v146: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:04.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:52:03 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:52:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:52:04.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:04 smithi007 ceph-mon[24826]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:04.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:04 smithi007 ceph-mon[29541]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:05.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:04 smithi033 ceph-mon[25651]: pgmap v147: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:06.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:05 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:06.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:05 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:06.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:05 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:07.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:06 smithi033 ceph-mon[25651]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:07.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:06 smithi007 ceph-mon[24826]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:07.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:06 smithi007 ceph-mon[29541]: pgmap v148: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:09.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:08 smithi033 ceph-mon[25651]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:09.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:08 smithi007 ceph-mon[24826]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:09.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:08 smithi007 ceph-mon[29541]: pgmap v149: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:11.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:10 smithi033 ceph-mon[25651]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:11.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:10 smithi007 ceph-mon[24826]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:11.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:10 smithi007 ceph-mon[29541]: pgmap v150: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:12.291 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:52:11 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:52:11] "GET /metrics HTTP/1.1" 200 214813 "" "Prometheus/2.33.4" 2024-04-15T21:52:13.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:12 smithi033 ceph-mon[25651]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:13.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:12 smithi007 ceph-mon[24826]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:13.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:12 smithi007 ceph-mon[29541]: pgmap v151: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:14.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:52:13 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:52:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:52:14.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:14 smithi007 ceph-mon[24826]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:14.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:14 smithi007 ceph-mon[29541]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:15.025 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:14 smithi033 ceph-mon[25651]: pgmap v152: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:16.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:15 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:16.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:15 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:16.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:15 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:17.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:16 smithi033 ceph-mon[25651]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:17.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:16 smithi007 ceph-mon[24826]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:17.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:16 smithi007 ceph-mon[29541]: pgmap v153: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:19.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:18 smithi033 ceph-mon[25651]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:19.041 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:18 smithi007 ceph-mon[24826]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:19.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:18 smithi007 ceph-mon[29541]: pgmap v154: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:21.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:20 smithi033 ceph-mon[25651]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:21.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:20 smithi007 ceph-mon[24826]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:21.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:20 smithi007 ceph-mon[29541]: pgmap v155: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:22.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:52:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:52:21] "GET /metrics HTTP/1.1" 200 214813 "" "Prometheus/2.33.4" 2024-04-15T21:52:23.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:22 smithi033 ceph-mon[25651]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:23.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:22 smithi007 ceph-mon[24826]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:23.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:22 smithi007 ceph-mon[29541]: pgmap v156: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:23.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-15T21:52:23.924 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:52:23 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:52:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:52:24.892 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:24 smithi007 ceph-mon[24826]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:24.892 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:24 smithi007 ceph-mon[29541]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:25.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:24 smithi033 ceph-mon[25651]: pgmap v157: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:25.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:25 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:25.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:25 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:26.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:25 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:26.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-15T21:52:26.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:26 smithi007 ceph-mon[24826]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:26.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:26 smithi007 ceph-mon[29541]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:27.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:26 smithi033 ceph-mon[25651]: pgmap v158: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:27.760 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:27 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:52:27.760 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:27 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:52:27.760 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:27 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:52:27.761 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:27 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:52:27.761 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:27 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:52:27.761 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:27 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:52:27.761 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:27 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:52:27.761 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:27 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:52:28.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:27 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:52:28.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:27 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:52:28.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:27 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:52:28.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:27 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:52:28.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-15T21:52:29.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:28 smithi033 ceph-mon[25651]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:29.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:28 smithi007 ceph-mon[24826]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:29.042 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:28 smithi007 ceph-mon[29541]: pgmap v159: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:30.969 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:30 smithi007 ceph-mon[24826]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:30.969 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:30 smithi007 ceph-mon[29541]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:31.022 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:30 smithi033 ceph-mon[25651]: pgmap v160: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:31.278 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-15T21:52:32.227 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:52:31 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:52:31] "GET /metrics HTTP/1.1" 200 214828 "" "Prometheus/2.33.4" 2024-04-15T21:52:32.769 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:32 smithi007 ceph-mon[24826]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:32.769 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:32 smithi007 ceph-mon[29541]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:33.023 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:32 smithi033 ceph-mon[25651]: pgmap v161: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:33.218 INFO:teuthology.orchestra.run.smithi007.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T21:52:33.831 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-15T21:52:33.841 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi007.front.sepia.ceph.com 2024-04-15T21:52:33.841 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-15T21:52:34.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:52:33 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:52:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:52:34.175 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:52:34.027+0000 7f564699f700 -1 log_channel(cephadm) log [ERR] : Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T21:52:34.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[24826]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:52:34.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[24826]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T21:52:34.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:34.427 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[24826]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[29541]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[29541]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:52:34.428 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:52:34.429 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:52:34.429 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:34.429 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[29541]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T21:52:34.429 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:34.429 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:52:34.429 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:52:34.429 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:52:34.429 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:34 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:34 smithi033 ceph-mon[25651]: from='client.24934 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:52:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:34 smithi033 ceph-mon[25651]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T21:52:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:34 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:34 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:52:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:34 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:52:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:34 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:52:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:34 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:34 smithi033 ceph-mon[25651]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T21:52:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:34 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:34 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:52:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:34 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:52:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:34 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:52:34.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:34 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:35.435 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:35 smithi007 ceph-mon[24826]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:35.435 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:35 smithi007 ceph-mon[24826]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T21:52:35.435 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:35 smithi007 ceph-mon[24826]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:35.435 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:35 smithi007 ceph-mon[24826]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-15T21:52:35.435 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:35 smithi007 ceph-mon[29541]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:35.435 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:35 smithi007 ceph-mon[29541]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T21:52:35.435 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:35 smithi007 ceph-mon[29541]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:35.436 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:35 smithi007 ceph-mon[29541]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-15T21:52:35.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:35 smithi033 ceph-mon[25651]: pgmap v162: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:35.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:35 smithi033 ceph-mon[25651]: Upgrade: Paused due to UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T21:52:35.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:35 smithi033 ceph-mon[25651]: pgmap v163: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:35.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:35 smithi033 ceph-mon[25651]: Health check failed: Upgrade: failed to pull target image (UPGRADE_FAILED_PULL) 2024-04-15T21:52:35.870 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-15T21:52:36.178 INFO:teuthology.orchestra.run.smithi007.stdout:"Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-15T21:52:36.473 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:36 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:36.474 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:36 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:36.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:36 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:36.763 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch ps' 2024-04-15T21:52:37.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:37 smithi007 ceph-mon[24826]: from='client.24940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:52:37.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:37 smithi007 ceph-mon[24826]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:37.394 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:37 smithi007 ceph-mon[24826]: from='client.24946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:52:37.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:37 smithi007 ceph-mon[29541]: from='client.24940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:52:37.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:37 smithi007 ceph-mon[29541]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:37.394 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:37 smithi007 ceph-mon[29541]: from='client.24946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:52:37.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:37 smithi033 ceph-mon[25651]: from='client.24940 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:52:37.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:37 smithi033 ceph-mon[25651]: pgmap v164: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:37.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:37 smithi033 ceph-mon[25651]: from='client.24946 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:52:38.659 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-15T21:52:38.659 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (2m) 117s ago 3m 15.5M - ba2b418f427c 9aff9a2b1756 2024-04-15T21:52:38.659 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi033 *:3000 running (2m) 117s ago 2m 45.5M - 8.3.5 dad864ee21e9 ba919d83187e 2024-04-15T21:52:38.659 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.phmsbn smithi007 running (2m) 117s ago 2m 74.7M - 3.5 e1d6a67b021e 32729c6fc1b3 2024-04-15T21:52:38.659 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi033 *:8443 running (7m) 117s ago 7m 411M - 17.2.0 e1d6a67b021e 6202e275d362 2024-04-15T21:52:38.660 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283 running (9m) 117s ago 9m 456M - 17.2.0 e1d6a67b021e 1f522423be81 2024-04-15T21:52:38.660 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (9m) 117s ago 9m 49.0M 2048M 17.2.0 e1d6a67b021e f3463cf07a92 2024-04-15T21:52:38.660 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi033 running (7m) 117s ago 7m 41.4M 2048M 17.2.0 e1d6a67b021e f5d803c8b945 2024-04-15T21:52:38.660 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (7m) 117s ago 7m 40.7M 2048M 17.2.0 e1d6a67b021e ee73c9e2f8d7 2024-04-15T21:52:38.660 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (3m) 117s ago 3m 16.5M - 1dbe0e931976 49d5ffb0b3d0 2024-04-15T21:52:38.660 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi033 *:9100 running (3m) 117s ago 3m 14.2M - 1dbe0e931976 80c30792c19b 2024-04-15T21:52:38.660 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (7m) 117s ago 7m 48.3M 4017M 17.2.0 e1d6a67b021e c3e6b4d48517 2024-04-15T21:52:38.660 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (6m) 117s ago 6m 49.3M 4017M 17.2.0 e1d6a67b021e 6cf5120d3adf 2024-04-15T21:52:38.660 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (6m) 117s ago 6m 44.9M 4017M 17.2.0 e1d6a67b021e 8f7ceb6d53f7 2024-04-15T21:52:38.660 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (6m) 117s ago 6m 48.8M 4017M 17.2.0 e1d6a67b021e 65be2d64ad47 2024-04-15T21:52:38.660 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi033 running (5m) 117s ago 5m 48.3M 4273M 17.2.0 e1d6a67b021e 632d3950f107 2024-04-15T21:52:38.660 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi033 running (5m) 117s ago 5m 45.8M 4273M 17.2.0 e1d6a67b021e eb2936b31d41 2024-04-15T21:52:38.660 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi033 running (4m) 117s ago 4m 45.2M 4273M 17.2.0 e1d6a67b021e 3b8a6a234cbb 2024-04-15T21:52:38.661 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi033 running (4m) 117s ago 4m 47.3M 4273M 17.2.0 e1d6a67b021e 2d7772ed29bc 2024-04-15T21:52:38.661 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi033 *:9095 running (2m) 117s ago 3m 44.1M - 514e6a882f6e 2102c8d49f6e 2024-04-15T21:52:38.661 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.qlizaq smithi007 *:8000 running (2m) 117s ago 2m 68.9M - 17.2.0 e1d6a67b021e d60e85ce3e4b 2024-04-15T21:52:38.661 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi033.glebqt smithi033 *:8000 running (2m) 117s ago 2m 70.6M - 17.2.0 e1d6a67b021e 691232f10203 2024-04-15T21:52:38.661 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.uhnuit smithi007 *:80 running (2m) 117s ago 2m 70.2M - 17.2.0 e1d6a67b021e 428ddd889110 2024-04-15T21:52:38.661 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi033.tenllr smithi033 *:80 running (2m) 117s ago 2m 71.0M - 17.2.0 e1d6a67b021e ff197c2f6873 2024-04-15T21:52:39.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph versions' 2024-04-15T21:52:39.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:39 smithi033 ceph-mon[25651]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:39.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:39 smithi007 ceph-mon[24826]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:39.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:39 smithi007 ceph-mon[29541]: pgmap v165: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:40.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:40 smithi033 ceph-mon[25651]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:52:40.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:40 smithi007 ceph-mon[24826]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:52:40.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:40 smithi007 ceph-mon[29541]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:52:41.278 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-15T21:52:41.278 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-15T21:52:41.278 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-15T21:52:41.278 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:52:41.278 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-15T21:52:41.278 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-15T21:52:41.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:52:41.279 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-15T21:52:41.279 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-15T21:52:41.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:52:41.279 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2024-04-15T21:52:41.279 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-15T21:52:41.279 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-15T21:52:41.279 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:52:41.279 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-15T21:52:41.279 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-15T21:52:41.279 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-15T21:52:41.279 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-15T21:52:41.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:41 smithi033 ceph-mon[25651]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:41.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:41 smithi007 ceph-mon[24826]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:41.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:41 smithi007 ceph-mon[29541]: pgmap v166: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:41.870 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:52:41 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:52:41] "GET /metrics HTTP/1.1" 200 214828 "" "Prometheus/2.33.4" 2024-04-15T21:52:41.893 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-04-15T21:52:42.507 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:42 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/3909172236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T21:52:42.507 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:42 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/3909172236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T21:52:42.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:42 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/3909172236' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T21:52:43.455 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:43 smithi007 ceph-mon[24826]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:43.455 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:43 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:43.455 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:43 smithi007 ceph-mon[29541]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:43.456 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:43 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:43.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:43 smithi033 ceph-mon[25651]: pgmap v167: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:43.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:43 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:52:43.555 INFO:teuthology.orchestra.run.smithi007.stdout:wait for servicemap items w/ changing names to refresh 2024-04-15T21:52:44.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:52:43 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:52:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:52:44.222 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'sleep 60' 2024-04-15T21:52:45.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:45 smithi033 ceph-mon[25651]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:45.532 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:45 smithi007 ceph-mon[24826]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:45.532 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:45 smithi007 ceph-mon[29541]: pgmap v168: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 958 B/s rd, 0 op/s 2024-04-15T21:52:46.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:46 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:46.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:46 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:46.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:46 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:47.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:47 smithi033 ceph-mon[25651]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:47.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:47 smithi007 ceph-mon[24826]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:47.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:47 smithi007 ceph-mon[29541]: pgmap v169: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:49.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:49 smithi033 ceph-mon[25651]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:49.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:49 smithi007 ceph-mon[24826]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:49.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:49 smithi007 ceph-mon[29541]: pgmap v170: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:50.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:50 smithi033 ceph-mon[25651]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:50.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:50 smithi007 ceph-mon[24826]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:50.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:50 smithi007 ceph-mon[29541]: pgmap v171: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:52.224 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:52:51 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:52:51] "GET /metrics HTTP/1.1" 200 214889 "" "Prometheus/2.33.4" 2024-04-15T21:52:53.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:53 smithi007 ceph-mon[24826]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:53.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:53 smithi007 ceph-mon[29541]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:53.476 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:53 smithi033 ceph-mon[25651]: pgmap v172: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:53.772 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:52:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:52:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:52:55.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:55 smithi033 ceph-mon[25651]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:55.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:55 smithi007 ceph-mon[24826]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:55.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:55 smithi007 ceph-mon[29541]: pgmap v173: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:52:56.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:56 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:56.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:56 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:56.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:56 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:52:57.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:57 smithi033 ceph-mon[25651]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:57.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:57 smithi007 ceph-mon[24826]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:57.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:57 smithi007 ceph-mon[29541]: pgmap v174: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:59.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:52:59 smithi033 ceph-mon[25651]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:59.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:52:59 smithi007 ceph-mon[24826]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:52:59.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:52:59 smithi007 ceph-mon[29541]: pgmap v175: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:00.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:00 smithi033 ceph-mon[25651]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:00.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:00 smithi007 ceph-mon[24826]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:00.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:00 smithi007 ceph-mon[29541]: pgmap v176: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:02.291 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:53:01 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:53:01] "GET /metrics HTTP/1.1" 200 214842 "" "Prometheus/2.33.4" 2024-04-15T21:53:03.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:03 smithi033 ceph-mon[25651]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:03.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:03 smithi007 ceph-mon[24826]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:03.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:03 smithi007 ceph-mon[29541]: pgmap v177: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:04.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:53:03 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:53:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:53:05.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:05 smithi033 ceph-mon[25651]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:05.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:05 smithi007 ceph-mon[24826]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:05.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:05 smithi007 ceph-mon[29541]: pgmap v178: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:06.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:06 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:06.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:06 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:06.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:06 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:07.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:07 smithi033 ceph-mon[25651]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:07.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:07 smithi007 ceph-mon[24826]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:07.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:07 smithi007 ceph-mon[29541]: pgmap v179: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:09.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:09 smithi033 ceph-mon[25651]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:09.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:09 smithi007 ceph-mon[24826]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:09.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:09 smithi007 ceph-mon[29541]: pgmap v180: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:10.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:10 smithi033 ceph-mon[25651]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:10.794 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:10 smithi007 ceph-mon[24826]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:10.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:10 smithi007 ceph-mon[29541]: pgmap v181: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:12.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:53:11 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:53:11] "GET /metrics HTTP/1.1" 200 214842 "" "Prometheus/2.33.4" 2024-04-15T21:53:13.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:13 smithi033 ceph-mon[25651]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:13.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:13 smithi007 ceph-mon[29541]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:13.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:13 smithi007 ceph-mon[24826]: pgmap v182: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:14.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:53:13 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:53:13] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:53:15.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:15 smithi033 ceph-mon[25651]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:15.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:15 smithi007 ceph-mon[29541]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:15.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:15 smithi007 ceph-mon[24826]: pgmap v183: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:16.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:16 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:16.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:16 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:16.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:16 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:17.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:17 smithi033 ceph-mon[25651]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:17.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:17 smithi007 ceph-mon[24826]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:17.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:17 smithi007 ceph-mon[29541]: pgmap v184: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:19.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:19 smithi033 ceph-mon[25651]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:19.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:19 smithi007 ceph-mon[24826]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:19.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:19 smithi007 ceph-mon[29541]: pgmap v185: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:20.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:20 smithi033 ceph-mon[25651]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:20.793 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:20 smithi007 ceph-mon[24826]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:20.794 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:20 smithi007 ceph-mon[29541]: pgmap v186: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:22.214 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:53:21 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:53:21] "GET /metrics HTTP/1.1" 200 214831 "" "Prometheus/2.33.4" 2024-04-15T21:53:23.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:23 smithi007 ceph-mon[24826]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:23.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:23 smithi007 ceph-mon[29541]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:23.476 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:23 smithi033 ceph-mon[25651]: pgmap v187: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:23.772 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:53:23 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:53:23] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:53:25.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:25 smithi033 ceph-mon[25651]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:25.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:25 smithi007 ceph-mon[24826]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:25.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:25 smithi007 ceph-mon[29541]: pgmap v188: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:26.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:26 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:26.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:26 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:26.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:26 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:27.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:27 smithi033 ceph-mon[25651]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:27.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:27 smithi007 ceph-mon[24826]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:27.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:27 smithi007 ceph-mon[29541]: pgmap v189: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:28.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:53:28.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:28 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:53:28.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:28 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:53:28.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:28 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:53:28.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:53:28.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:28 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:53:28.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:28 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:53:28.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:28 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:53:28.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:53:28.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:28 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:53:28.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:28 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-04-15T21:53:28.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:28 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-04-15T21:53:29.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:29 smithi033 ceph-mon[25651]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:29.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:29 smithi007 ceph-mon[24826]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:29.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:29 smithi007 ceph-mon[29541]: pgmap v190: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:30.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:30 smithi033 ceph-mon[25651]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:30.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:30 smithi007 ceph-mon[24826]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:30.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:30 smithi007 ceph-mon[29541]: pgmap v191: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:32.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:53:31 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:53:31] "GET /metrics HTTP/1.1" 200 214830 "" "Prometheus/2.33.4" 2024-04-15T21:53:33.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:33 smithi033 ceph-mon[25651]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:33.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:33 smithi007 ceph-mon[24826]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:33.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:33 smithi007 ceph-mon[29541]: pgmap v192: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:34.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:53:33 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:53:33] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:53:34.388 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:34 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:53:34.389 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:34 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:53:34.389 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:34 smithi033 ceph-mon[25651]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:53:34.390 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:34 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:53:34.390 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:34 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:53:34.391 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:34 smithi007 ceph-mon[29541]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:53:34.391 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:34 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-15T21:53:34.391 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:34 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-15T21:53:34.391 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:34 smithi007 ceph-mon[24826]: from='mgr.24358 172.21.15.7:0/1183343540' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-15T21:53:35.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:35 smithi033 ceph-mon[25651]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:35.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:35 smithi033 ceph-mon[25651]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:53:35.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:35 smithi007 ceph-mon[24826]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:35.542 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:35 smithi007 ceph-mon[24826]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:53:35.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:35 smithi007 ceph-mon[29541]: pgmap v193: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:35.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:35 smithi007 ceph-mon[29541]: from='mgr.24358 ' entity='mgr.y' 2024-04-15T21:53:36.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:36 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:36.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:36 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:36.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:36 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:37.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:37 smithi033 ceph-mon[25651]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:37.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:37 smithi007 ceph-mon[24826]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:37.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:37 smithi007 ceph-mon[29541]: pgmap v194: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:39.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:39 smithi033 ceph-mon[25651]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:39.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:39 smithi007 ceph-mon[24826]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:39.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:39 smithi007 ceph-mon[29541]: pgmap v195: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:40.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:40 smithi033 ceph-mon[25651]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:40.792 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:40 smithi007 ceph-mon[24826]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:40.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:40 smithi007 ceph-mon[29541]: pgmap v196: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:42.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:53:41 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:53:41] "GET /metrics HTTP/1.1" 200 214830 "" "Prometheus/2.33.4" 2024-04-15T21:53:43.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:43 smithi033 ceph-mon[25651]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:43.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:43 smithi007 ceph-mon[24826]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:43.541 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:43 smithi007 ceph-mon[29541]: pgmap v197: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:44.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:53:43 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:53:43] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:53:45.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:45 smithi033 ceph-mon[25651]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:45.541 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:45 smithi007 ceph-mon[24826]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:45.542 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:45 smithi007 ceph-mon[29541]: pgmap v198: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:46.428 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:46 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:46.428 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:46 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:46.462 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch ps' 2024-04-15T21:53:46.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:46 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:47.467 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:47 smithi007 ceph-mon[24826]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:47.467 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:47 smithi007 ceph-mon[29541]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:47.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:47 smithi033 ceph-mon[25651]: pgmap v199: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:48.370 INFO:teuthology.orchestra.run.smithi007.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-15T21:53:48.371 INFO:teuthology.orchestra.run.smithi007.stdout:alertmanager.a smithi007 *:9093,9094 running (3m) 3m ago 4m 15.5M - ba2b418f427c 9aff9a2b1756 2024-04-15T21:53:48.371 INFO:teuthology.orchestra.run.smithi007.stdout:grafana.a smithi033 *:3000 running (3m) 3m ago 3m 45.5M - 8.3.5 dad864ee21e9 ba919d83187e 2024-04-15T21:53:48.371 INFO:teuthology.orchestra.run.smithi007.stdout:iscsi.foo.smithi007.phmsbn smithi007 running (3m) 3m ago 3m 74.7M - 3.5 e1d6a67b021e 32729c6fc1b3 2024-04-15T21:53:48.371 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.x smithi033 *:8443 running (8m) 3m ago 8m 411M - 17.2.0 e1d6a67b021e 6202e275d362 2024-04-15T21:53:48.371 INFO:teuthology.orchestra.run.smithi007.stdout:mgr.y smithi007 *:9283 running (10m) 3m ago 10m 456M - 17.2.0 e1d6a67b021e 1f522423be81 2024-04-15T21:53:48.371 INFO:teuthology.orchestra.run.smithi007.stdout:mon.a smithi007 running (10m) 3m ago 10m 49.0M 2048M 17.2.0 e1d6a67b021e f3463cf07a92 2024-04-15T21:53:48.371 INFO:teuthology.orchestra.run.smithi007.stdout:mon.b smithi033 running (8m) 3m ago 8m 41.4M 2048M 17.2.0 e1d6a67b021e f5d803c8b945 2024-04-15T21:53:48.371 INFO:teuthology.orchestra.run.smithi007.stdout:mon.c smithi007 running (9m) 3m ago 9m 40.7M 2048M 17.2.0 e1d6a67b021e ee73c9e2f8d7 2024-04-15T21:53:48.371 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.a smithi007 *:9100 running (4m) 3m ago 4m 16.5M - 1dbe0e931976 49d5ffb0b3d0 2024-04-15T21:53:48.371 INFO:teuthology.orchestra.run.smithi007.stdout:node-exporter.b smithi033 *:9100 running (4m) 3m ago 4m 14.2M - 1dbe0e931976 80c30792c19b 2024-04-15T21:53:48.371 INFO:teuthology.orchestra.run.smithi007.stdout:osd.0 smithi007 running (8m) 3m ago 8m 48.3M 4017M 17.2.0 e1d6a67b021e c3e6b4d48517 2024-04-15T21:53:48.371 INFO:teuthology.orchestra.run.smithi007.stdout:osd.1 smithi007 running (8m) 3m ago 8m 49.3M 4017M 17.2.0 e1d6a67b021e 6cf5120d3adf 2024-04-15T21:53:48.371 INFO:teuthology.orchestra.run.smithi007.stdout:osd.2 smithi007 running (7m) 3m ago 7m 44.9M 4017M 17.2.0 e1d6a67b021e 8f7ceb6d53f7 2024-04-15T21:53:48.372 INFO:teuthology.orchestra.run.smithi007.stdout:osd.3 smithi007 running (7m) 3m ago 7m 48.8M 4017M 17.2.0 e1d6a67b021e 65be2d64ad47 2024-04-15T21:53:48.372 INFO:teuthology.orchestra.run.smithi007.stdout:osd.4 smithi033 running (6m) 3m ago 6m 48.3M 4273M 17.2.0 e1d6a67b021e 632d3950f107 2024-04-15T21:53:48.372 INFO:teuthology.orchestra.run.smithi007.stdout:osd.5 smithi033 running (6m) 3m ago 6m 45.8M 4273M 17.2.0 e1d6a67b021e eb2936b31d41 2024-04-15T21:53:48.372 INFO:teuthology.orchestra.run.smithi007.stdout:osd.6 smithi033 running (6m) 3m ago 6m 45.2M 4273M 17.2.0 e1d6a67b021e 3b8a6a234cbb 2024-04-15T21:53:48.372 INFO:teuthology.orchestra.run.smithi007.stdout:osd.7 smithi033 running (5m) 3m ago 5m 47.3M 4273M 17.2.0 e1d6a67b021e 2d7772ed29bc 2024-04-15T21:53:48.372 INFO:teuthology.orchestra.run.smithi007.stdout:prometheus.a smithi033 *:9095 running (3m) 3m ago 5m 44.1M - 514e6a882f6e 2102c8d49f6e 2024-04-15T21:53:48.372 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi007.qlizaq smithi007 *:8000 running (3m) 3m ago 3m 68.9M - 17.2.0 e1d6a67b021e d60e85ce3e4b 2024-04-15T21:53:48.372 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.foo.smithi033.glebqt smithi033 *:8000 running (3m) 3m ago 3m 70.6M - 17.2.0 e1d6a67b021e 691232f10203 2024-04-15T21:53:48.372 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi007.uhnuit smithi007 *:80 running (3m) 3m ago 3m 70.2M - 17.2.0 e1d6a67b021e 428ddd889110 2024-04-15T21:53:48.372 INFO:teuthology.orchestra.run.smithi007.stdout:rgw.smpl.smithi033.tenllr smithi033 *:80 running (3m) 3m ago 3m 71.0M - 17.2.0 e1d6a67b021e ff197c2f6873 2024-04-15T21:53:49.007 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph versions' 2024-04-15T21:53:49.262 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:49 smithi007 ceph-mon[24826]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:49.262 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:49 smithi007 ceph-mon[29541]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:49.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:49 smithi033 ceph-mon[25651]: pgmap v200: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:50.299 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:50 smithi007 ceph-mon[24826]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:53:50.299 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:50 smithi007 ceph-mon[29541]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:53:50.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:50 smithi033 ceph-mon[25651]: from='client.15060 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:53:50.881 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-15T21:53:50.881 INFO:teuthology.orchestra.run.smithi007.stdout: "mon": { 2024-04-15T21:53:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3 2024-04-15T21:53:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:53:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "mgr": { 2024-04-15T21:53:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-15T21:53:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:53:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "osd": { 2024-04-15T21:53:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-15T21:53:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:53:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "mds": {}, 2024-04-15T21:53:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "rgw": { 2024-04-15T21:53:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4 2024-04-15T21:53:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: }, 2024-04-15T21:53:50.882 INFO:teuthology.orchestra.run.smithi007.stdout: "overall": { 2024-04-15T21:53:50.883 INFO:teuthology.orchestra.run.smithi007.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 17 2024-04-15T21:53:50.883 INFO:teuthology.orchestra.run.smithi007.stdout: } 2024-04-15T21:53:50.883 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-15T21:53:51.291 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:51 smithi007 ceph-mon[24826]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:51.292 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:51 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/4123112235' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T21:53:51.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:51 smithi007 ceph-mon[29541]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:51.292 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:51 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/4123112235' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T21:53:51.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:51 smithi033 ceph-mon[25651]: pgmap v201: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:51.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:51 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/4123112235' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T21:53:51.616 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph orch upgrade status' 2024-04-15T21:53:51.932 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:53:51 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:53:51] "GET /metrics HTTP/1.1" 200 214833 "" "Prometheus/2.33.4" 2024-04-15T21:53:53.201 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:53 smithi007 ceph-mon[24826]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:53.201 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:53 smithi007 ceph-mon[29541]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:53.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:53 smithi033 ceph-mon[25651]: pgmap v202: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:53.616 INFO:teuthology.orchestra.run.smithi007.stdout:{ 2024-04-15T21:53:53.616 INFO:teuthology.orchestra.run.smithi007.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765", 2024-04-15T21:53:53.616 INFO:teuthology.orchestra.run.smithi007.stdout: "in_progress": true, 2024-04-15T21:53:53.616 INFO:teuthology.orchestra.run.smithi007.stdout: "services_complete": [], 2024-04-15T21:53:53.616 INFO:teuthology.orchestra.run.smithi007.stdout: "progress": "", 2024-04-15T21:53:53.616 INFO:teuthology.orchestra.run.smithi007.stdout: "message": "Error: UPGRADE_FAILED_PULL: Upgrade: failed to pull target image" 2024-04-15T21:53:53.616 INFO:teuthology.orchestra.run.smithi007.stdout:} 2024-04-15T21:53:54.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:53:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:53:53] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:53:54.275 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph health detail' 2024-04-15T21:53:55.379 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:55 smithi007 ceph-mon[24826]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:53:55.379 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:55 smithi007 ceph-mon[24826]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:55.379 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:55 smithi007 ceph-mon[29541]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:53:55.379 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:55 smithi007 ceph-mon[29541]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:55.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:55 smithi033 ceph-mon[25651]: from='client.15072 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-15T21:53:55.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:55 smithi033 ceph-mon[25651]: pgmap v203: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:53:56.156 INFO:teuthology.orchestra.run.smithi007.stdout:HEALTH_WARN Upgrade: failed to pull target image 2024-04-15T21:53:56.156 INFO:teuthology.orchestra.run.smithi007.stdout:[WRN] UPGRADE_FAILED_PULL: Upgrade: failed to pull target image 2024-04-15T21:53:56.156 INFO:teuthology.orchestra.run.smithi007.stdout: host smithi007 `cephadm pull` failed: cephadm exited with an error code: 1, stderr: Pulling container image quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765... 2024-04-15T21:53:56.156 INFO:teuthology.orchestra.run.smithi007.stdout:Non-zero exit code 125 from /bin/podman pull quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T21:53:56.156 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765... 2024-04-15T21:53:56.156 INFO:teuthology.orchestra.run.smithi007.stdout:/bin/podman: stderr Error: initializing source docker://quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765: reading manifest a9a752df26c63acad72e1b3569fd79a515ca0765 in quay.ceph.io/ceph-ci/ceph: manifest unknown 2024-04-15T21:53:56.156 INFO:teuthology.orchestra.run.smithi007.stdout:ERROR: Failed command: /bin/podman pull quay.ceph.io/ceph-ci/ceph:a9a752df26c63acad72e1b3569fd79a515ca0765 2024-04-15T21:53:56.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:56 smithi033 ceph-mon[25651]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:56.535 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:56 smithi007 ceph-mon[24826]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:56.536 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:56 smithi007 ceph-mon[29541]: from='client.24775 -' entity='client.iscsi.foo.smithi007.phmsbn' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-04-15T21:53:56.952 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-04-15T21:53:57.267 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:57 smithi007 ceph-mon[24826]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:57.267 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:57 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1928228574' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-15T21:53:57.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:57 smithi007 ceph-mon[29541]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:57.267 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:57 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1928228574' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-15T21:53:57.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:57 smithi033 ceph-mon[25651]: pgmap v204: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:57.522 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:57 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1928228574' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-15T21:53:58.892 INFO:teuthology.orchestra.run.smithi007.stdout:true 2024-04-15T21:53:59.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:59 smithi007 ceph-mon[24826]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:59.287 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:53:59 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/1864582697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T21:53:59.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:59 smithi007 ceph-mon[29541]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:59.287 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:53:59 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/1864582697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T21:53:59.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:59 smithi033 ceph-mon[25651]: pgmap v205: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:53:59.523 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:53:59 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/1864582697' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T21:53:59.559 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-04-15T21:54:00.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:00 smithi033 ceph-mon[25651]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:54:00.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:54:00 smithi007 ceph-mon[24826]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:54:00.791 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:54:00 smithi007 ceph-mon[29541]: pgmap v206: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-04-15T21:54:01.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:01 smithi033 ceph-mon[25651]: from='client.? 172.21.15.7:0/339616982' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T21:54:01.791 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:54:01 smithi007 ceph-mon[24826]: from='client.? 172.21.15.7:0/339616982' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T21:54:01.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:54:01 smithi007 ceph-mon[29541]: from='client.? 172.21.15.7:0/339616982' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-15T21:54:02.165 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:54:01 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:54:01] "GET /metrics HTTP/1.1" 200 214846 "" "Prometheus/2.33.4" 2024-04-15T21:54:02.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T21:54:02.190 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-15T21:54:02.389 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=2f4d5abd2f49463591127c9a995a0444 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_461a84e6fbbf089011eafd13b98169fb9e9b8cc9/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c \'ceph versions | jq -e \'"\'"\'.overall | keys\'"\'"\' | grep $sha1\'' 2024-04-15T21:54:02.392 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-15T21:54:02.401 INFO:tasks.cephadm:Teardown begin 2024-04-15T21:54:02.402 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T21:54:02.439 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T21:54:02.474 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-15T21:54:02.474 DEBUG:teuthology.orchestra.run.smithi007:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-15T21:54:02.495 DEBUG:teuthology.orchestra.run.smithi033:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-15T21:54:02.531 INFO:tasks.cephadm:Stopping all daemons... 2024-04-15T21:54:02.532 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-15T21:54:02.532 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.a 2024-04-15T21:54:02.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:54:02 smithi007 ceph-mon[24826]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:54:02.719 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:54:02 smithi007 systemd[1]: Stopping Ceph mon.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:54:02.720 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:54:02 smithi007 bash[57191]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon.a" found: no such container 2024-04-15T21:54:02.720 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:54:02 smithi007 ceph-mon[29541]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:54:02.772 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:02 smithi033 ceph-mon[25651]: pgmap v207: 161 pgs: 161 active+clean; 585 KiB data, 94 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-04-15T21:54:03.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:54:02 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a[24822]: 2024-04-15T21:54:02.717+0000 7f7c536fd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T21:54:03.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:54:02 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a[24822]: 2024-04-15T21:54:02.717+0000 7f7c536fd700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-15T21:54:03.042 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:54:02 smithi007 podman[57204]: 2024-04-15 21:54:02.871408882 +0000 UTC m=+0.254369120 container died f3463cf07a9237a4fbc6c53496166800dcfcb18e04896afe06467374e4e64f3f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870) 2024-04-15T21:54:03.503 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:54:03 smithi007 podman[57204]: 2024-04-15 21:54:03.301237476 +0000 UTC m=+0.684197720 container cleanup f3463cf07a9237a4fbc6c53496166800dcfcb18e04896afe06467374e4e64f3f (image=quay.io/ceph/ceph:v17.2.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a, distribution-scope=public, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, version=8, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, GIT_CLEAN=True) 2024-04-15T21:54:03.504 INFO:journalctl@ceph.mon.a.smithi007.stdout:Apr 15 21:54:03 smithi007 bash[57204]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-a 2024-04-15T21:54:03.525 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.a.service' 2024-04-15T21:54:03.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:54:03.593 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-15T21:54:03.594 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-04-15T21:54:03.594 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.c 2024-04-15T21:54:03.791 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:54:03 smithi007 systemd[1]: Stopping Ceph mon.c for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:54:03.792 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:54:03 smithi007 bash[57284]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon.c" found: no such container 2024-04-15T21:54:04.022 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:54:03 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x[26830]: ::ffff:172.21.15.33 - - [15/Apr/2024:21:54:03] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.33.4" 2024-04-15T21:54:04.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:54:03 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-c[29537]: 2024-04-15T21:54:03.796+0000 7f5134da1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T21:54:04.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:54:03 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-c[29537]: 2024-04-15T21:54:03.796+0000 7f5134da1700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-15T21:54:04.074 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:54:03 smithi007 podman[57296]: 2024-04-15 21:54:03.818028018 +0000 UTC m=+0.142977009 container died ee73c9e2f8d75b1c6ce0f7080f2e8f8741d2da412531170b2df570c0b7629fa0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-c, io.openshift.expose-services=, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, vcs-type=git, version=8, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, maintainer=Guillaume Abrioux , ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, name=centos-stream) 2024-04-15T21:54:04.408 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:54:04 smithi007 podman[57296]: 2024-04-15 21:54:04.24762986 +0000 UTC m=+0.572578893 container cleanup ee73c9e2f8d75b1c6ce0f7080f2e8f8741d2da412531170b2df570c0b7629fa0 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-c, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_CLEAN=True, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, release=754, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, version=8, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-15T21:54:04.408 INFO:journalctl@ceph.mon.c.smithi007.stdout:Apr 15 21:54:04 smithi007 bash[57296]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-c 2024-04-15T21:54:04.474 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.c.service' 2024-04-15T21:54:04.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:54:04.517 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-04-15T21:54:04.517 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-04-15T21:54:04.517 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.b 2024-04-15T21:54:04.798 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:04 smithi033 systemd[1]: Stopping Ceph mon.b for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:54:04.798 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:04 smithi033 bash[46493]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon.b" found: no such container 2024-04-15T21:54:04.798 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:04 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-b[25635]: 2024-04-15T21:54:04.713+0000 7ff582a37700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T21:54:04.798 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:04 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-b[25635]: 2024-04-15T21:54:04.713+0000 7ff582a37700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-15T21:54:05.207 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:04 smithi033 podman[46504]: 2024-04-15 21:54:04.796987629 +0000 UTC m=+0.194226312 container died f5d803c8b945f1e57f66466add6390db6777963b7379919e4ff4f3eb0b9aa1a7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-b, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, name=centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, ceph=True, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.openshift.expose-services=, release=754, version=8, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-15T21:54:05.456 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.b.service' 2024-04-15T21:54:05.484 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:05 smithi033 podman[46504]: 2024-04-15 21:54:05.206165386 +0000 UTC m=+0.603404114 container cleanup f5d803c8b945f1e57f66466add6390db6777963b7379919e4ff4f3eb0b9aa1a7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-b, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, version=8, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_CLEAN=True, architecture=x86_64, com.redhat.component=centos-stream-container, vcs-type=git, RELEASE=HEAD, name=centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, release=754) 2024-04-15T21:54:05.484 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:05 smithi033 bash[46504]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-b 2024-04-15T21:54:05.484 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:05 smithi033 bash[46531]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon.b" found: no such container 2024-04-15T21:54:05.485 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:05 smithi033 podman[46518]: 2024-04-15 21:54:05.449247621 +0000 UTC m=+0.650754540 container remove f5d803c8b945f1e57f66466add6390db6777963b7379919e4ff4f3eb0b9aa1a7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mon-b, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, release=754, RELEASE=HEAD, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , architecture=x86_64, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, ceph=True, vcs-type=git, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-15T21:54:05.485 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:05 smithi033 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.b.service: Deactivated successfully. 2024-04-15T21:54:05.485 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:05 smithi033 systemd[1]: Stopped Ceph mon.b for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:54:05.485 INFO:journalctl@ceph.mon.b.smithi033.stdout:Apr 15 21:54:05 smithi033 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mon.b.service: Consumed 6.851s CPU time. 2024-04-15T21:54:05.506 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:54:05.507 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-04-15T21:54:05.507 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-15T21:54:05.507 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mgr.y 2024-04-15T21:54:05.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:54:05 smithi007 systemd[1]: Stopping Ceph mgr.y for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:54:05.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:54:05 smithi007 bash[57436]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr.y" found: no such container 2024-04-15T21:54:05.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:54:05 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y[25056]: 2024-04-15T21:54:05.709+0000 7f56b25ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T21:54:05.792 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:54:05 smithi007 podman[57446]: 2024-04-15 21:54:05.741218511 +0000 UTC m=+0.146501425 container died 1f522423be818b9bcb61c71d114e8e36f05f5025062ecdc418e81c62872c0f66 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y, GIT_CLEAN=True, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, version=8, release=754, maintainer=Guillaume Abrioux , RELEASE=HEAD, distribution-scope=public, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True) 2024-04-15T21:54:06.291 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:54:06 smithi007 podman[57446]: 2024-04-15 21:54:06.101890646 +0000 UTC m=+0.507173561 container cleanup 1f522423be818b9bcb61c71d114e8e36f05f5025062ecdc418e81c62872c0f66 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, ceph=True, distribution-scope=public, vendor=Red Hat, Inc., vcs-type=git, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, version=8, GIT_BRANCH=HEAD, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-15T21:54:06.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:54:06 smithi007 bash[57446]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-y 2024-04-15T21:54:06.292 INFO:journalctl@ceph.mgr.y.smithi007.stdout:Apr 15 21:54:06 smithi007 bash[57472]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr.y" found: no such container 2024-04-15T21:54:06.384 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mgr.y.service' 2024-04-15T21:54:06.428 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:54:06.428 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-15T21:54:06.428 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-15T21:54:06.428 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mgr.x 2024-04-15T21:54:06.773 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:54:06 smithi033 systemd[1]: Stopping Ceph mgr.x for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:54:06.773 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:54:06 smithi033 bash[46586]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr.x" found: no such container 2024-04-15T21:54:06.773 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:54:06 smithi033 podman[46599]: 2024-04-15 21:54:06.648601278 +0000 UTC m=+0.139188733 container died 6202e275d3623c349623fdb6c5bb2c30e572497bdf5460edbbd5ae094505e962 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x, architecture=x86_64, version=8, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, com.redhat.component=centos-stream-container, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, release=754, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, vcs-type=git) 2024-04-15T21:54:07.263 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:54:07 smithi033 podman[46599]: 2024-04-15 21:54:07.036114101 +0000 UTC m=+0.526701624 container cleanup 6202e275d3623c349623fdb6c5bb2c30e572497bdf5460edbbd5ae094505e962 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, distribution-scope=public, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, release=754, vcs-type=git, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, ceph=True, GIT_BRANCH=HEAD, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True) 2024-04-15T21:54:07.263 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:54:07 smithi033 bash[46599]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr-x 2024-04-15T21:54:07.263 INFO:journalctl@ceph.mgr.x.smithi033.stdout:Apr 15 21:54:07 smithi033 bash[46625]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-mgr.x" found: no such container 2024-04-15T21:54:07.268 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@mgr.x.service' 2024-04-15T21:54:07.310 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:54:07.311 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-15T21:54:07.311 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-15T21:54:07.312 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.0 2024-04-15T21:54:07.791 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:07 smithi007 systemd[1]: Stopping Ceph osd.0 for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:54:07.792 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:07 smithi007 bash[57527]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T21:54:07.792 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:07 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0[32406]: 2024-04-15T21:54:07.480+0000 7fd070ddb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T21:54:07.792 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:07 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0[32406]: 2024-04-15T21:54:07.480+0000 7fd070ddb700 -1 osd.0 96 *** Got signal Terminated *** 2024-04-15T21:54:07.792 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:07 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0[32406]: 2024-04-15T21:54:07.480+0000 7fd070ddb700 -1 osd.0 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T21:54:13.233 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:12 smithi007 podman[57540]: 2024-04-15 21:54:12.940805226 +0000 UTC m=+5.546327186 container died c3e6b4d485178a2f1a48f722a449a6e1db1e94740a881099dd39557db7fbbf89 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, com.redhat.component=centos-stream-container, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, distribution-scope=public, release=754, vendor=Red Hat, Inc.) 2024-04-15T21:54:13.542 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:13 smithi007 podman[57540]: 2024-04-15 21:54:13.372345199 +0000 UTC m=+5.977867160 container cleanup c3e6b4d485178a2f1a48f722a449a6e1db1e94740a881099dd39557db7fbbf89 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0, ceph=True, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, vcs-type=git, vendor=Red Hat, Inc., io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, release=754, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , name=centos-stream, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-15T21:54:13.542 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:13 smithi007 bash[57540]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0 2024-04-15T21:54:13.542 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:13 smithi007 bash[57569]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.0" found: no such container 2024-04-15T21:54:13.952 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:13 smithi007 podman[57557]: 2024-04-15 21:54:13.66537509 +0000 UTC m=+0.718187735 container remove c3e6b4d485178a2f1a48f722a449a6e1db1e94740a881099dd39557db7fbbf89 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_CLEAN=True, RELEASE=HEAD, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vcs-type=git, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, version=8, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-15T21:54:13.952 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:13 smithi007 podman[57634]: 2024-04-15 21:54:13.764674382 +0000 UTC m=+0.019358910 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T21:54:13.952 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:13 smithi007 podman[57634]: 2024-04-15 21:54:13.866351984 +0000 UTC m=+0.121036515 container create 13e1365c7dcd71aac2d298ba7dd3f1df708a8e8179974acb57ba42ac67d2948a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, maintainer=Guillaume Abrioux , version=8, RELEASE=HEAD, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, release=754, distribution-scope=public) 2024-04-15T21:54:14.292 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:14 smithi007 podman[57634]: 2024-04-15 21:54:14.118897921 +0000 UTC m=+0.373582460 container init 13e1365c7dcd71aac2d298ba7dd3f1df708a8e8179974acb57ba42ac67d2948a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., GIT_CLEAN=True, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, io.openshift.expose-services=, com.redhat.component=centos-stream-container, RELEASE=HEAD, architecture=x86_64, io.buildah.version=1.19.8, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, name=centos-stream, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-15T21:54:14.292 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:14 smithi007 podman[57634]: 2024-04-15 21:54:14.122210938 +0000 UTC m=+0.376895465 container start 13e1365c7dcd71aac2d298ba7dd3f1df708a8e8179974acb57ba42ac67d2948a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., vcs-type=git, GIT_CLEAN=True, distribution-scope=public, architecture=x86_64, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-15T21:54:14.292 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:14 smithi007 podman[57634]: 2024-04-15 21:54:14.193723364 +0000 UTC m=+0.448407892 container attach 13e1365c7dcd71aac2d298ba7dd3f1df708a8e8179974acb57ba42ac67d2948a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0-deactivate, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, build-date=2022-05-03T08:36:31.336870, name=centos-stream, ceph=True, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, vcs-type=git, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-15T21:54:14.697 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:14 smithi007 podman[57634]: 2024-04-15 21:54:14.409395297 +0000 UTC m=+0.664079842 container died 13e1365c7dcd71aac2d298ba7dd3f1df708a8e8179974acb57ba42ac67d2948a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, vendor=Red Hat, Inc., ceph=True, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, RELEASE=HEAD, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, distribution-scope=public) 2024-04-15T21:54:14.948 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:14 smithi007 podman[57660]: 2024-04-15 21:54:14.930240552 +0000 UTC m=+0.514874699 container remove 13e1365c7dcd71aac2d298ba7dd3f1df708a8e8179974acb57ba42ac67d2948a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , name=centos-stream, version=8, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, ceph=True, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, GIT_CLEAN=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-15T21:54:14.948 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:14 smithi007 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.0.service: Deactivated successfully. 2024-04-15T21:54:14.948 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:14 smithi007 systemd[1]: Stopped Ceph osd.0 for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:54:14.948 INFO:journalctl@ceph.osd.0.smithi007.stdout:Apr 15 21:54:14 smithi007 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.0.service: Consumed 5.277s CPU time. 2024-04-15T21:54:14.950 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.0.service' 2024-04-15T21:54:14.994 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:54:14.994 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-15T21:54:14.995 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-15T21:54:14.995 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.1 2024-04-15T21:54:15.217 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:15 smithi007 systemd[1]: Stopping Ceph osd.1 for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:54:15.218 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:15 smithi007 bash[57713]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T21:54:15.541 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:15 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1[35375]: 2024-04-15T21:54:15.215+0000 7f51357fa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T21:54:15.542 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:15 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1[35375]: 2024-04-15T21:54:15.215+0000 7f51357fa700 -1 osd.1 96 *** Got signal Terminated *** 2024-04-15T21:54:15.542 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:15 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1[35375]: 2024-04-15T21:54:15.215+0000 7f51357fa700 -1 osd.1 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T21:54:20.903 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:20 smithi007 podman[57726]: 2024-04-15 21:54:20.649737127 +0000 UTC m=+5.543138531 container died 6cf5120d3adf5691b2a6bd1e18ec621bc864e52e7aae5662cbd6bb41fb577670 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , version=8, io.buildah.version=1.19.8, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, RELEASE=HEAD, distribution-scope=public, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-15T21:54:21.256 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:21 smithi007 podman[57726]: 2024-04-15 21:54:21.066268651 +0000 UTC m=+5.959670054 container cleanup 6cf5120d3adf5691b2a6bd1e18ec621bc864e52e7aae5662cbd6bb41fb577670 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, build-date=2022-05-03T08:36:31.336870, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, architecture=x86_64, GIT_CLEAN=True, vcs-type=git, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , distribution-scope=public, RELEASE=HEAD) 2024-04-15T21:54:21.257 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:21 smithi007 bash[57726]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1 2024-04-15T21:54:21.257 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:21 smithi007 bash[57759]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.1" found: no such container 2024-04-15T21:54:21.542 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:21 smithi007 podman[57746]: 2024-04-15 21:54:21.401283371 +0000 UTC m=+0.746230508 container remove 6cf5120d3adf5691b2a6bd1e18ec621bc864e52e7aae5662cbd6bb41fb577670 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, io.buildah.version=1.19.8, vendor=Red Hat, Inc., RELEASE=HEAD, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, name=centos-stream, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, maintainer=Guillaume Abrioux , distribution-scope=public) 2024-04-15T21:54:21.930 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:21 smithi007 podman[57831]: 2024-04-15 21:54:21.551486772 +0000 UTC m=+0.018674569 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T21:54:21.931 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:21 smithi007 podman[57831]: 2024-04-15 21:54:21.660503309 +0000 UTC m=+0.127691090 container create bee013854227c62c3cb8c025ce893cae18b1140fa7fed73baa623bd8b5066bb6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1-deactivate, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8) 2024-04-15T21:54:21.931 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:21 smithi007 podman[57831]: 2024-04-15 21:54:21.871125545 +0000 UTC m=+0.338313326 container init bee013854227c62c3cb8c025ce893cae18b1140fa7fed73baa623bd8b5066bb6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1-deactivate, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, architecture=x86_64, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, version=8, name=centos-stream, ceph=True) 2024-04-15T21:54:21.931 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:21 smithi007 podman[57831]: 2024-04-15 21:54:21.874488957 +0000 UTC m=+0.341676749 container start bee013854227c62c3cb8c025ce893cae18b1140fa7fed73baa623bd8b5066bb6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1-deactivate, vcs-type=git, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., name=centos-stream, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , architecture=x86_64, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD) 2024-04-15T21:54:21.931 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:21 smithi007 podman[57831]: 2024-04-15 21:54:21.92936356 +0000 UTC m=+0.396551349 container attach bee013854227c62c3cb8c025ce893cae18b1140fa7fed73baa623bd8b5066bb6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1-deactivate, com.redhat.component=centos-stream-container, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, ceph=True, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vcs-type=git, version=8) 2024-04-15T21:54:22.192 INFO:journalctl@ceph.osd.1.smithi007.stdout:Apr 15 21:54:22 smithi007 podman[57831]: 2024-04-15 21:54:22.168941524 +0000 UTC m=+0.636129305 container died bee013854227c62c3cb8c025ce893cae18b1140fa7fed73baa623bd8b5066bb6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-1-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, io.openshift.expose-services=, release=754, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, ceph=True, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-15T21:54:22.770 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.1.service' 2024-04-15T21:54:22.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:54:22.813 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-15T21:54:22.813 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-15T21:54:22.813 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.2 2024-04-15T21:54:23.020 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:22 smithi007 systemd[1]: Stopping Ceph osd.2 for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:54:23.021 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:22 smithi007 bash[57909]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T21:54:23.291 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:23 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2[38306]: 2024-04-15T21:54:23.019+0000 7f24ff59c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T21:54:23.291 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:23 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2[38306]: 2024-04-15T21:54:23.019+0000 7f24ff59c700 -1 osd.2 96 *** Got signal Terminated *** 2024-04-15T21:54:23.292 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:23 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2[38306]: 2024-04-15T21:54:23.019+0000 7f24ff59c700 -1 osd.2 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T21:54:28.792 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:28 smithi007 podman[57920]: 2024-04-15 21:54:28.422681112 +0000 UTC m=+5.497227928 container died 8f7ceb6d53f74ee9f250f030b78fa9921e0169ab51d3c092eef9a0ca7dcc69cc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., version=8, name=centos-stream, RELEASE=HEAD, ceph=True, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_CLEAN=True, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-15T21:54:29.246 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:28 smithi007 podman[57920]: 2024-04-15 21:54:28.804702559 +0000 UTC m=+5.879249426 container cleanup 8f7ceb6d53f74ee9f250f030b78fa9921e0169ab51d3c092eef9a0ca7dcc69cc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, com.redhat.component=centos-stream-container, RELEASE=HEAD, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, name=centos-stream, architecture=x86_64, vendor=Red Hat, Inc., version=8, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_BRANCH=HEAD, distribution-scope=public, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-15T21:54:29.246 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:28 smithi007 bash[57920]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2 2024-04-15T21:54:29.246 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:28 smithi007 bash[57950]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.2" found: no such container 2024-04-15T21:54:29.246 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:29 smithi007 podman[57938]: 2024-04-15 21:54:29.045000981 +0000 UTC m=+0.616414168 container remove 8f7ceb6d53f74ee9f250f030b78fa9921e0169ab51d3c092eef9a0ca7dcc69cc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, release=754, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, vcs-type=git, io.openshift.expose-services=, vendor=Red Hat, Inc., ceph=True, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD) 2024-04-15T21:54:29.541 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:29 smithi007 podman[58015]: 2024-04-15 21:54:29.148429142 +0000 UTC m=+0.018713983 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T21:54:29.542 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:29 smithi007 podman[58015]: 2024-04-15 21:54:29.245678357 +0000 UTC m=+0.115963197 container create 181d05e46db3a6d2344ede15e4c2b054858f8685c6f2ad626c88e6fdc3bc2e26 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2-deactivate, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, name=centos-stream, vcs-type=git, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., ceph=True, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8) 2024-04-15T21:54:29.542 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:29 smithi007 podman[58015]: 2024-04-15 21:54:29.473399231 +0000 UTC m=+0.343684080 container init 181d05e46db3a6d2344ede15e4c2b054858f8685c6f2ad626c88e6fdc3bc2e26 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2-deactivate, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., version=8, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, release=754, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux ) 2024-04-15T21:54:29.542 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:29 smithi007 podman[58015]: 2024-04-15 21:54:29.47674217 +0000 UTC m=+0.347027013 container start 181d05e46db3a6d2344ede15e4c2b054858f8685c6f2ad626c88e6fdc3bc2e26 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2-deactivate, vcs-type=git, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, ceph=True, io.openshift.expose-services=, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, name=centos-stream, io.buildah.version=1.19.8, RELEASE=HEAD) 2024-04-15T21:54:30.041 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:29 smithi007 podman[58015]: 2024-04-15 21:54:29.548252302 +0000 UTC m=+0.418537157 container attach 181d05e46db3a6d2344ede15e4c2b054858f8685c6f2ad626c88e6fdc3bc2e26 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2-deactivate, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, release=754, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, io.buildah.version=1.19.8, GIT_CLEAN=True, RELEASE=HEAD, architecture=x86_64, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True) 2024-04-15T21:54:30.042 INFO:journalctl@ceph.osd.2.smithi007.stdout:Apr 15 21:54:29 smithi007 podman[58015]: 2024-04-15 21:54:29.769779417 +0000 UTC m=+0.640064260 container died 181d05e46db3a6d2344ede15e4c2b054858f8685c6f2ad626c88e6fdc3bc2e26 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, version=8, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.expose-services=, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, vendor=Red Hat, Inc.) 2024-04-15T21:54:30.442 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.2.service' 2024-04-15T21:54:30.486 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:54:30.487 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-15T21:54:30.487 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-15T21:54:30.487 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.3 2024-04-15T21:54:30.664 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:30 smithi007 systemd[1]: Stopping Ceph osd.3 for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:54:30.665 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:30 smithi007 bash[58094]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T21:54:31.041 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:30 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3[41290]: 2024-04-15T21:54:30.663+0000 7fd8b619f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T21:54:31.042 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:30 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3[41290]: 2024-04-15T21:54:30.663+0000 7fd8b619f700 -1 osd.3 96 *** Got signal Terminated *** 2024-04-15T21:54:31.042 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:30 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3[41290]: 2024-04-15T21:54:30.663+0000 7fd8b619f700 -1 osd.3 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T21:54:34.523 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:34 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:34.199+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:35.022 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:34 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:34.741+0000 7fcbf5f6f700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:10.313262+0000 front 2024-04-15T21:54:10.313196+0000 (oldest deadline 2024-04-15T21:54:34.413111+0000) 2024-04-15T21:54:35.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:34 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:34.531+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:35.292 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:34 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3[41290]: 2024-04-15T21:54:34.918+0000 7fd8b1337700 -1 osd.3 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.787031+0000 front 2024-04-15T21:54:09.787147+0000 (oldest deadline 2024-04-15T21:54:34.487153+0000) 2024-04-15T21:54:35.484 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:35 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:35.201+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:35.772 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:35 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:35.712+0000 7fcbf5f6f700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:10.313262+0000 front 2024-04-15T21:54:10.313196+0000 (oldest deadline 2024-04-15T21:54:34.413111+0000) 2024-04-15T21:54:35.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:35 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:35.514+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:36.446 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:36 smithi007 podman[58108]: 2024-04-15 21:54:36.046934545 +0000 UTC m=+5.483529619 container died 65be2d64ad474e48ab2b2cabf1511b6979bfb6c960c1e312d5449a0a621a6955 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3, architecture=x86_64, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, version=8, distribution-scope=public, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, RELEASE=HEAD, ceph=True) 2024-04-15T21:54:36.446 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:36 smithi007 podman[58108]: 2024-04-15 21:54:36.445503038 +0000 UTC m=+5.882098096 container cleanup 65be2d64ad474e48ab2b2cabf1511b6979bfb6c960c1e312d5449a0a621a6955 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3, io.openshift.tags=base centos centos-stream, distribution-scope=public, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, CEPH_POINT_RELEASE=-17.2.0, version=8, maintainer=Guillaume Abrioux , release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, com.redhat.component=centos-stream-container, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870) 2024-04-15T21:54:36.522 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:36 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:36.190+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:36.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:36 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:36.108+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:36.792 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:36 smithi007 bash[58108]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3 2024-04-15T21:54:36.792 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:36 smithi007 bash[58140]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.3" found: no such container 2024-04-15T21:54:36.792 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:36 smithi007 podman[58126]: 2024-04-15 21:54:36.655313708 +0000 UTC m=+0.604964701 container remove 65be2d64ad474e48ab2b2cabf1511b6979bfb6c960c1e312d5449a0a621a6955 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, version=8, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, io.buildah.version=1.19.8, ceph=True, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, architecture=x86_64) 2024-04-15T21:54:37.022 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:36 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:36.676+0000 7fcbf5f6f700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:10.313262+0000 front 2024-04-15T21:54:10.313196+0000 (oldest deadline 2024-04-15T21:54:34.413111+0000) 2024-04-15T21:54:37.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:36 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:36.527+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:37.168 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:36 smithi007 podman[58201]: 2024-04-15 21:54:36.880950823 +0000 UTC m=+0.114278136 container create b3023c1fc094ce8e460918f7531a64d99a6152a1ef20196b2d0b01e18bd15c81 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3-deactivate, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.openshift.expose-services=, vcs-type=git, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, version=8, CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, architecture=x86_64, release=754) 2024-04-15T21:54:37.168 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:36 smithi007 podman[58201]: 2024-04-15 21:54:36.785573116 +0000 UTC m=+0.018900439 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T21:54:37.168 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:37 smithi007 podman[58201]: 2024-04-15 21:54:37.108288299 +0000 UTC m=+0.341615613 container init b3023c1fc094ce8e460918f7531a64d99a6152a1ef20196b2d0b01e18bd15c81 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3-deactivate, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , RELEASE=HEAD, distribution-scope=public, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.openshift.expose-services=, ceph=True, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-15T21:54:37.168 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:37 smithi007 podman[58201]: 2024-04-15 21:54:37.111645788 +0000 UTC m=+0.344973112 container start b3023c1fc094ce8e460918f7531a64d99a6152a1ef20196b2d0b01e18bd15c81 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, distribution-scope=public, vendor=Red Hat, Inc., release=754, com.redhat.component=centos-stream-container, ceph=True, name=centos-stream, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, io.buildah.version=1.19.8, vcs-type=git, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-15T21:54:37.168 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:37 smithi007 podman[58201]: 2024-04-15 21:54:37.166617985 +0000 UTC m=+0.399945326 container attach b3023c1fc094ce8e460918f7531a64d99a6152a1ef20196b2d0b01e18bd15c81 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3-deactivate, architecture=x86_64, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.openshift.expose-services=, RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, ceph=True, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True) 2024-04-15T21:54:37.522 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:37 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:37.229+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:37.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:37 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:37.125+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:37.542 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:37 smithi007 podman[58201]: 2024-04-15 21:54:37.399130974 +0000 UTC m=+0.632458293 container died b3023c1fc094ce8e460918f7531a64d99a6152a1ef20196b2d0b01e18bd15c81 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, RELEASE=HEAD, com.redhat.component=centos-stream-container, distribution-scope=public, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, name=centos-stream, architecture=x86_64, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, GIT_BRANCH=HEAD, version=8) 2024-04-15T21:54:37.890 INFO:journalctl@ceph.osd.3.smithi007.stdout:Apr 15 21:54:37 smithi007 podman[58228]: 2024-04-15 21:54:37.861256245 +0000 UTC m=+0.452667883 container remove b3023c1fc094ce8e460918f7531a64d99a6152a1ef20196b2d0b01e18bd15c81 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-3-deactivate, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vcs-type=git, architecture=x86_64, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., RELEASE=HEAD, io.openshift.expose-services=, release=754, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, version=8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-15T21:54:37.894 DEBUG:teuthology.orchestra.run.smithi007:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.3.service' 2024-04-15T21:54:37.936 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:54:37.936 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-15T21:54:37.937 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-15T21:54:37.937 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.4 2024-04-15T21:54:37.969 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:37 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:37.543+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:37.970 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:37 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:37.725+0000 7fcbf5f6f700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:10.313262+0000 front 2024-04-15T21:54:10.313196+0000 (oldest deadline 2024-04-15T21:54:34.413111+0000) 2024-04-15T21:54:38.245 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:38 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:38.108+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:38.245 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:37 smithi033 systemd[1]: Stopping Ceph osd.4 for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:54:38.245 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:38 smithi033 bash[46694]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T21:54:38.245 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:38 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:38.129+0000 7fcbfc5da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T21:54:38.245 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:38 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:38.129+0000 7fcbfc5da700 -1 osd.4 96 *** Got signal Terminated *** 2024-04-15T21:54:38.246 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:38 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:38.129+0000 7fcbfc5da700 -1 osd.4 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T21:54:38.522 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:38 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:38.243+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:39.022 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:38 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:38.713+0000 7fcbf5f6f700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:10.313262+0000 front 2024-04-15T21:54:10.313196+0000 (oldest deadline 2024-04-15T21:54:34.413111+0000) 2024-04-15T21:54:39.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:38 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:38.544+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:39.522 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:39 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:39.194+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:39.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:39 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:39.143+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:40.022 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:39 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:39.757+0000 7fcbf5f6f700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:10.313262+0000 front 2024-04-15T21:54:10.313196+0000 (oldest deadline 2024-04-15T21:54:34.413111+0000) 2024-04-15T21:54:40.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:39 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:39.531+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:40.522 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:40 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:40.206+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:40.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:40 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:40.102+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:41.022 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:40 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:40.775+0000 7fcbf5f6f700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:10.313262+0000 front 2024-04-15T21:54:10.313196+0000 (oldest deadline 2024-04-15T21:54:34.413111+0000) 2024-04-15T21:54:41.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:40 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:40.533+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:41.522 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:41 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:41.143+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:41.523 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:41 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:41.256+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:41.523 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:41 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:41.256+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.147098+0000 front 2024-04-15T21:54:19.147232+0000 (oldest deadline 2024-04-15T21:54:40.847052+0000) 2024-04-15T21:54:42.022 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:41 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:41.529+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:42.023 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:41 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:41.746+0000 7fcbf5f6f700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:10.313262+0000 front 2024-04-15T21:54:10.313196+0000 (oldest deadline 2024-04-15T21:54:34.413111+0000) 2024-04-15T21:54:42.023 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:41 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:41.746+0000 7fcbf5f6f700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:15.514099+0000 front 2024-04-15T21:54:15.514016+0000 (oldest deadline 2024-04-15T21:54:40.814187+0000) 2024-04-15T21:54:42.499 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:42 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:42.186+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:42.499 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:42 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:42.299+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:42.500 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:42 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:42.299+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.147098+0000 front 2024-04-15T21:54:19.147232+0000 (oldest deadline 2024-04-15T21:54:40.847052+0000) 2024-04-15T21:54:42.772 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:42 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:42.498+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:43.177 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:42 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:42.780+0000 7fcbf5f6f700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:10.313262+0000 front 2024-04-15T21:54:10.313196+0000 (oldest deadline 2024-04-15T21:54:34.413111+0000) 2024-04-15T21:54:43.178 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:42 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4[29036]: 2024-04-15T21:54:42.780+0000 7fcbf5f6f700 -1 osd.4 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:15.514099+0000 front 2024-04-15T21:54:15.514016+0000 (oldest deadline 2024-04-15T21:54:40.814187+0000) 2024-04-15T21:54:43.492 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:43 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:43.282+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:43.492 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:43 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:43.282+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.147098+0000 front 2024-04-15T21:54:19.147232+0000 (oldest deadline 2024-04-15T21:54:40.847052+0000) 2024-04-15T21:54:43.493 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:43 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:43.176+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:43.493 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:43 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:43.176+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:43.773 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:43 smithi033 podman[46707]: 2024-04-15 21:54:43.551593707 +0000 UTC m=+5.522104229 container died 632d3950f107127eec48b027c97e3f88401b433caab20a1e802eecec337e514d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, vcs-type=git, architecture=x86_64, name=centos-stream, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_CLEAN=True, ceph=True, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-15T21:54:43.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:43 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:43.491+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:43.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:43 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:43.491+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:44.097 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:43 smithi033 podman[46707]: 2024-04-15 21:54:43.895228183 +0000 UTC m=+5.865738693 container cleanup 632d3950f107127eec48b027c97e3f88401b433caab20a1e802eecec337e514d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., io.openshift.expose-services=, release=754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, maintainer=Guillaume Abrioux , RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, GIT_CLEAN=True) 2024-04-15T21:54:44.097 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:43 smithi033 bash[46707]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4 2024-04-15T21:54:44.400 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:44.245+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:44.400 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:44.245+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.147098+0000 front 2024-04-15T21:54:19.147232+0000 (oldest deadline 2024-04-15T21:54:40.847052+0000) 2024-04-15T21:54:44.401 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:44 smithi033 podman[46724]: 2024-04-15 21:54:44.096465032 +0000 UTC m=+0.546995319 container remove 632d3950f107127eec48b027c97e3f88401b433caab20a1e802eecec337e514d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, release=754, architecture=x86_64, io.openshift.expose-services=, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, ceph=True, version=8, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-15T21:54:44.401 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:44 smithi033 bash[46740]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.4" found: no such container 2024-04-15T21:54:44.401 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:44 smithi033 podman[46803]: 2024-04-15 21:54:44.305603626 +0000 UTC m=+0.092101972 container create 587e417fa95a41c11655606bb1e2c21107d238b9b1f774faf5cd4eda1c6f6edc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, version=8, com.redhat.component=centos-stream-container, vcs-type=git, architecture=x86_64, vendor=Red Hat, Inc., release=754, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, distribution-scope=public, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-15T21:54:44.401 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:44 smithi033 podman[46803]: 2024-04-15 21:54:44.232193812 +0000 UTC m=+0.018692185 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T21:54:44.401 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:44.145+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:44.402 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:44.145+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:44.772 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:44.460+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:44.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:44 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:44.460+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:44.773 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:44 smithi033 podman[46803]: 2024-04-15 21:54:44.516556648 +0000 UTC m=+0.303054993 container init 587e417fa95a41c11655606bb1e2c21107d238b9b1f774faf5cd4eda1c6f6edc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4-deactivate, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.openshift.tags=base centos centos-stream, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, architecture=x86_64, vendor=Red Hat, Inc., GIT_CLEAN=True, ceph=True, io.buildah.version=1.19.8) 2024-04-15T21:54:44.773 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:44 smithi033 podman[46803]: 2024-04-15 21:54:44.519998352 +0000 UTC m=+0.306496700 container start 587e417fa95a41c11655606bb1e2c21107d238b9b1f774faf5cd4eda1c6f6edc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4-deactivate, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, name=centos-stream, ceph=True, vcs-type=git, release=754, GIT_BRANCH=HEAD, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8) 2024-04-15T21:54:44.774 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:44 smithi033 podman[46803]: 2024-04-15 21:54:44.59124792 +0000 UTC m=+0.377746299 container attach 587e417fa95a41c11655606bb1e2c21107d238b9b1f774faf5cd4eda1c6f6edc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-type=git, architecture=x86_64, vendor=Red Hat, Inc., version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_CLEAN=True, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-15T21:54:45.095 INFO:journalctl@ceph.osd.4.smithi033.stdout:Apr 15 21:54:44 smithi033 podman[46803]: 2024-04-15 21:54:44.815298469 +0000 UTC m=+0.601796824 container died 587e417fa95a41c11655606bb1e2c21107d238b9b1f774faf5cd4eda1c6f6edc (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, ceph=True, name=centos-stream, vcs-type=git, GIT_BRANCH=HEAD, RELEASE=HEAD, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, io.buildah.version=1.19.8, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_CLEAN=True, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-15T21:54:45.379 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:45 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:45.280+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:45.379 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:45 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:45.280+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.147098+0000 front 2024-04-15T21:54:19.147232+0000 (oldest deadline 2024-04-15T21:54:40.847052+0000) 2024-04-15T21:54:45.379 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:45 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:45.143+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:45.380 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:45 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:45.143+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:45.401 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.4.service' 2024-04-15T21:54:45.484 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:54:45.484 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-15T21:54:45.485 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-15T21:54:45.485 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.5 2024-04-15T21:54:45.708 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:45 smithi033 systemd[1]: Stopping Ceph osd.5 for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:54:45.708 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:45 smithi033 bash[46883]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T21:54:45.709 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:45 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:45.493+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:45.709 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:45 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:45.493+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:46.022 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:45 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:45.706+0000 7f5cecc3d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T21:54:46.022 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:45 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:45.706+0000 7f5cecc3d700 -1 osd.5 96 *** Got signal Terminated *** 2024-04-15T21:54:46.023 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:45 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:45.706+0000 7f5cecc3d700 -1 osd.5 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T21:54:46.459 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:46 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:46.112+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:46.459 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:46 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:46.112+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:46.460 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:46 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:46.302+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:46.460 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:46 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:46.302+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.147098+0000 front 2024-04-15T21:54:19.147232+0000 (oldest deadline 2024-04-15T21:54:40.847052+0000) 2024-04-15T21:54:46.772 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:46 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:46.458+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:46.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:46 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:46.458+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:47.428 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:47 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:47.281+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:47.428 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:47 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:47.281+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.147098+0000 front 2024-04-15T21:54:19.147232+0000 (oldest deadline 2024-04-15T21:54:40.847052+0000) 2024-04-15T21:54:47.429 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:47 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:47.105+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:47.429 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:47 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:47.105+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:47.772 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:47 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:47.426+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:47.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:47 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:47.426+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:48.440 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:48 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:48.232+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:48.440 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:48 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:48.232+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.147098+0000 front 2024-04-15T21:54:19.147232+0000 (oldest deadline 2024-04-15T21:54:40.847052+0000) 2024-04-15T21:54:48.440 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:48 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:48.142+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:48.440 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:48 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:48.142+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:48.772 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:48 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:48.438+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:48.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:48 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:48.438+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:49.449 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:49 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:49.184+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:49.449 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:49 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:49.184+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.147098+0000 front 2024-04-15T21:54:19.147232+0000 (oldest deadline 2024-04-15T21:54:40.847052+0000) 2024-04-15T21:54:49.450 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:49 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:49.184+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:49.450 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:49 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:49.184+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:49.450 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:49 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:49.184+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:54:49.772 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:49 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:49.447+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:49.772 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:49 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:49.447+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:50.488 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:50 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:50.179+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:09.146369+0000 front 2024-04-15T21:54:09.146258+0000 (oldest deadline 2024-04-15T21:54:33.846170+0000) 2024-04-15T21:54:50.488 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:50 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5[31985]: 2024-04-15T21:54:50.179+0000 7f5ce7dd5700 -1 osd.5 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.147098+0000 front 2024-04-15T21:54:19.147232+0000 (oldest deadline 2024-04-15T21:54:40.847052+0000) 2024-04-15T21:54:50.488 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:50 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:50.158+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:50.488 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:50 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:50.158+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:50.488 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:50 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:50.158+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:54:50.772 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:50 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:50.486+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:50.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:50 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:50.486+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:50.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:50 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:50.486+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:23.065525+0000 front 2024-04-15T21:54:23.065542+0000 (oldest deadline 2024-04-15T21:54:49.466267+0000) 2024-04-15T21:54:51.407 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:51 smithi033 podman[46894]: 2024-04-15 21:54:51.025774362 +0000 UTC m=+5.425987470 container died eb2936b31d41c168780b24895ccc12fc3be5f21f27618c92000573526040ec51 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., distribution-scope=public, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, vcs-type=git, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, version=8, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , release=754, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-15T21:54:51.407 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:51 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:51.205+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:51.408 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:51 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:51.205+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:51.408 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:51 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:51.205+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:54:51.772 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:51 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:51.533+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:51.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:51 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:51.533+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:51.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:51 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:51.533+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:23.065525+0000 front 2024-04-15T21:54:23.065542+0000 (oldest deadline 2024-04-15T21:54:49.466267+0000) 2024-04-15T21:54:51.773 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:51 smithi033 podman[46894]: 2024-04-15 21:54:51.406288253 +0000 UTC m=+5.806501389 container cleanup eb2936b31d41c168780b24895ccc12fc3be5f21f27618c92000573526040ec51 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, name=centos-stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, vcs-type=git, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, release=754) 2024-04-15T21:54:51.773 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:51 smithi033 bash[46894]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5 2024-04-15T21:54:51.773 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:51 smithi033 bash[46930]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.5" found: no such container 2024-04-15T21:54:51.774 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:51 smithi033 podman[46915]: 2024-04-15 21:54:51.649128569 +0000 UTC m=+0.619986023 container remove eb2936b31d41c168780b24895ccc12fc3be5f21f27618c92000573526040ec51 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, ceph=True, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, version=8, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., release=754, GIT_CLEAN=True, name=centos-stream, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git) 2024-04-15T21:54:52.103 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:51 smithi033 podman[46992]: 2024-04-15 21:54:51.751946475 +0000 UTC m=+0.019511624 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T21:54:52.103 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:51 smithi033 podman[46992]: 2024-04-15 21:54:51.875208588 +0000 UTC m=+0.142773739 container create 934fce6a21f90af3c7b202d783dd411e7e011af245fccb1408b2f3cfff458e22 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5-deactivate, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, distribution-scope=public, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, vcs-type=git) 2024-04-15T21:54:52.104 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:52 smithi033 podman[46992]: 2024-04-15 21:54:52.102602798 +0000 UTC m=+0.370167947 container init 934fce6a21f90af3c7b202d783dd411e7e011af245fccb1408b2f3cfff458e22 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., ceph=True, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, io.buildah.version=1.19.8, name=centos-stream, io.openshift.expose-services=, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-15T21:54:52.396 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:52 smithi033 podman[46992]: 2024-04-15 21:54:52.106281044 +0000 UTC m=+0.373846193 container start 934fce6a21f90af3c7b202d783dd411e7e011af245fccb1408b2f3cfff458e22 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, name=centos-stream, io.openshift.expose-services=, vendor=Red Hat, Inc., version=8, com.redhat.component=centos-stream-container, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, io.k8s.display-name=CentOS Stream 8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-15T21:54:52.397 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:52 smithi033 podman[46992]: 2024-04-15 21:54:52.194297669 +0000 UTC m=+0.461862833 container attach 934fce6a21f90af3c7b202d783dd411e7e011af245fccb1408b2f3cfff458e22 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, version=8, maintainer=Guillaume Abrioux , name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, ceph=True, distribution-scope=public, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-15T21:54:52.397 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:52 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:52.234+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:52.397 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:52 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:52.234+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:52.397 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:52 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:52.234+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:54:52.681 INFO:journalctl@ceph.osd.5.smithi033.stdout:Apr 15 21:54:52 smithi033 podman[46992]: 2024-04-15 21:54:52.395999144 +0000 UTC m=+0.663564299 container died 934fce6a21f90af3c7b202d783dd411e7e011af245fccb1408b2f3cfff458e22 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-5-deactivate, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, distribution-scope=public, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, RELEASE=HEAD, vendor=Red Hat, Inc., version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, name=centos-stream, com.redhat.component=centos-stream-container) 2024-04-15T21:54:52.681 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:52 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:52.559+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:52.682 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:52 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:52.559+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:52.682 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:52 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:52.559+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:23.065525+0000 front 2024-04-15T21:54:23.065542+0000 (oldest deadline 2024-04-15T21:54:49.466267+0000) 2024-04-15T21:54:53.016 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.5.service' 2024-04-15T21:54:53.062 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:54:53.062 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-15T21:54:53.062 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-15T21:54:53.063 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.6 2024-04-15T21:54:53.273 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:53 smithi033 systemd[1]: Stopping Ceph osd.6 for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:54:53.273 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:53 smithi033 bash[47069]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T21:54:53.273 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:53.213+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:53.273 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:53.213+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:53.274 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:53.213+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:54:53.610 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:53.317+0000 7ff9ba6bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T21:54:53.611 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:53.317+0000 7ff9ba6bb700 -1 osd.6 96 *** Got signal Terminated *** 2024-04-15T21:54:53.611 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:53.317+0000 7ff9ba6bb700 -1 osd.6 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T21:54:54.022 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:53.608+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:54.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:53.608+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:54.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:53 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:53.608+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:23.065525+0000 front 2024-04-15T21:54:23.065542+0000 (oldest deadline 2024-04-15T21:54:49.466267+0000) 2024-04-15T21:54:54.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:54.256+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:54.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:54.256+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:54.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:54.256+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:54:55.022 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:54.628+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:55.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:54.628+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:55.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:54 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:54.628+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:23.065525+0000 front 2024-04-15T21:54:23.065542+0000 (oldest deadline 2024-04-15T21:54:49.466267+0000) 2024-04-15T21:54:55.522 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:55 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:55.244+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:55.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:55 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:55.244+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:55.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:55 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:55.244+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:54:56.022 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:55 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:55.616+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:56.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:55 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:55.616+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:56.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:55 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:55.616+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:23.065525+0000 front 2024-04-15T21:54:23.065542+0000 (oldest deadline 2024-04-15T21:54:49.466267+0000) 2024-04-15T21:54:56.522 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:56 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:56.281+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:56.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:56 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:56.281+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:56.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:56 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:56.281+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:54:57.022 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:56 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:56.606+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:57.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:56 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:56.606+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:57.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:56 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:56.606+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:23.065525+0000 front 2024-04-15T21:54:23.065542+0000 (oldest deadline 2024-04-15T21:54:49.466267+0000) 2024-04-15T21:54:57.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:56 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:56.606+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-15T21:54:35.267478+0000 front 2024-04-15T21:54:35.267558+0000 (oldest deadline 2024-04-15T21:54:56.367331+0000) 2024-04-15T21:54:57.648 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:57 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:57.286+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:57.649 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:57 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:57.286+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:57.649 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:57 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:57.286+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:54:58.022 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:57 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:57.646+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:07.863486+0000 front 2024-04-15T21:54:07.863637+0000 (oldest deadline 2024-04-15T21:54:33.763532+0000) 2024-04-15T21:54:58.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:57 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:57.646+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:19.565182+0000 front 2024-04-15T21:54:19.565152+0000 (oldest deadline 2024-04-15T21:54:43.065053+0000) 2024-04-15T21:54:58.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:57 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:57.646+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:23.065525+0000 front 2024-04-15T21:54:23.065542+0000 (oldest deadline 2024-04-15T21:54:49.466267+0000) 2024-04-15T21:54:58.023 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:57 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6[34863]: 2024-04-15T21:54:57.646+0000 7ff9b384f700 -1 osd.6 96 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-15T21:54:35.267478+0000 front 2024-04-15T21:54:35.267558+0000 (oldest deadline 2024-04-15T21:54:56.367331+0000) 2024-04-15T21:54:58.740 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:58 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:58.320+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:58.740 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:58 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:58.320+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:58.740 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:58 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:58.320+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:54:59.022 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:58 smithi033 podman[47082]: 2024-04-15 21:54:58.739010229 +0000 UTC m=+5.555588704 container died 3b8a6a234cbb822ffdc91575a3096243baff402c9f26bbb64644b793765d03b6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6, ceph=True, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, distribution-scope=public, RELEASE=HEAD, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-15T21:54:59.511 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:59 smithi033 podman[47082]: 2024-04-15 21:54:59.067230455 +0000 UTC m=+5.883808927 container cleanup 3b8a6a234cbb822ffdc91575a3096243baff402c9f26bbb64644b793765d03b6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, ceph=True, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, distribution-scope=public, name=centos-stream) 2024-04-15T21:54:59.511 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:59 smithi033 bash[47082]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6 2024-04-15T21:54:59.511 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:59 smithi033 podman[47101]: 2024-04-15 21:54:59.23482104 +0000 UTC m=+0.494963527 container remove 3b8a6a234cbb822ffdc91575a3096243baff402c9f26bbb64644b793765d03b6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, version=8, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, maintainer=Guillaume Abrioux , release=754, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public) 2024-04-15T21:54:59.511 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:59 smithi033 bash[47117]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.6" found: no such container 2024-04-15T21:54:59.512 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:59 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:59.306+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:54:59.512 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:59 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:59.306+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:54:59.512 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:54:59 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:54:59.306+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:54:59.772 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:59 smithi033 podman[47181]: 2024-04-15 21:54:59.413805097 +0000 UTC m=+0.019691410 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T21:54:59.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:59 smithi033 podman[47181]: 2024-04-15 21:54:59.510459714 +0000 UTC m=+0.116346024 container create 78ef74c37005bcc2dd16f2a05b7f75027f16dc6d7554315f594f52d3d7cd21c5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6-deactivate, io.buildah.version=1.19.8, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, vcs-type=git, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, RELEASE=HEAD, GIT_CLEAN=True, distribution-scope=public, release=754) 2024-04-15T21:54:59.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:59 smithi033 podman[47181]: 2024-04-15 21:54:59.737858236 +0000 UTC m=+0.343744561 container init 78ef74c37005bcc2dd16f2a05b7f75027f16dc6d7554315f594f52d3d7cd21c5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6-deactivate, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, architecture=x86_64, version=8, RELEASE=HEAD, name=centos-stream, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD) 2024-04-15T21:54:59.773 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:59 smithi033 podman[47181]: 2024-04-15 21:54:59.74141169 +0000 UTC m=+0.347298006 container start 78ef74c37005bcc2dd16f2a05b7f75027f16dc6d7554315f594f52d3d7cd21c5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6-deactivate, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, io.openshift.expose-services=, ceph=True, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.openshift.tags=base centos centos-stream) 2024-04-15T21:55:00.264 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:54:59 smithi033 podman[47181]: 2024-04-15 21:54:59.821337333 +0000 UTC m=+0.427223646 container attach 78ef74c37005bcc2dd16f2a05b7f75027f16dc6d7554315f594f52d3d7cd21c5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, GIT_BRANCH=HEAD, vcs-type=git, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8) 2024-04-15T21:55:00.289 INFO:journalctl@ceph.osd.6.smithi033.stdout:Apr 15 21:55:00 smithi033 podman[47181]: 2024-04-15 21:55:00.031645708 +0000 UTC m=+0.637532014 container died 78ef74c37005bcc2dd16f2a05b7f75027f16dc6d7554315f594f52d3d7cd21c5 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-6-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.buildah.version=1.19.8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, distribution-scope=public, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, release=754, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, vendor=Red Hat, Inc.) 2024-04-15T21:55:00.522 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:00 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:00.263+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:55:00.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:00 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:00.263+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:55:00.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:00 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:00.263+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:55:00.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:00 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:00.263+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-15T21:54:34.265916+0000 front 2024-04-15T21:54:34.265850+0000 (oldest deadline 2024-04-15T21:54:59.565735+0000) 2024-04-15T21:55:00.624 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.6.service' 2024-04-15T21:55:00.668 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:55:00.668 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-15T21:55:00.669 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-15T21:55:00.669 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.7 2024-04-15T21:55:00.895 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:00 smithi033 systemd[1]: Stopping Ceph osd.7 for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:55:00.896 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:00 smithi033 bash[47260]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T21:55:00.896 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:00 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:00.893+0000 7f3345e9e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-15T21:55:00.896 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:00 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:00.893+0000 7f3345e9e700 -1 osd.7 96 *** Got signal Terminated *** 2024-04-15T21:55:00.896 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:00 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:00.893+0000 7f3345e9e700 -1 osd.7 96 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-15T21:55:01.773 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:01 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:01.284+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:55:01.773 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:01 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:01.284+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:55:01.773 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:01 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:01.284+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:55:01.773 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:01 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:01.284+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-15T21:54:34.265916+0000 front 2024-04-15T21:54:34.265850+0000 (oldest deadline 2024-04-15T21:54:59.565735+0000) 2024-04-15T21:55:02.772 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:02 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:02.319+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:55:02.773 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:02 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:02.319+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:55:02.773 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:02 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:02.319+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:55:02.773 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:02 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:02.319+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-15T21:54:34.265916+0000 front 2024-04-15T21:54:34.265850+0000 (oldest deadline 2024-04-15T21:54:59.565735+0000) 2024-04-15T21:55:03.772 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:03 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:03.334+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:55:03.773 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:03 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:03.334+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:55:03.773 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:03 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:03.334+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:55:03.773 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:03 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:03.334+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-15T21:54:34.265916+0000 front 2024-04-15T21:54:34.265850+0000 (oldest deadline 2024-04-15T21:54:59.565735+0000) 2024-04-15T21:55:04.674 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:04 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:04.287+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:55:04.674 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:04 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:04.287+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:55:04.674 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:04 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:04.287+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:55:04.674 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:04 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:04.287+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-15T21:54:34.265916+0000 front 2024-04-15T21:54:34.265850+0000 (oldest deadline 2024-04-15T21:54:59.565735+0000) 2024-04-15T21:55:05.522 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:05 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:05.269+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6806 osd.0 since back 2024-04-15T21:54:12.262136+0000 front 2024-04-15T21:54:12.262195+0000 (oldest deadline 2024-04-15T21:54:35.662362+0000) 2024-04-15T21:55:05.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:05 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:05.269+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6814 osd.1 since back 2024-04-15T21:54:17.363451+0000 front 2024-04-15T21:54:17.363551+0000 (oldest deadline 2024-04-15T21:54:42.663409+0000) 2024-04-15T21:55:05.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:05 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:05.269+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6822 osd.2 since back 2024-04-15T21:54:22.663968+0000 front 2024-04-15T21:54:22.663991+0000 (oldest deadline 2024-04-15T21:54:48.563998+0000) 2024-04-15T21:55:05.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:05 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:05.269+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.7:6830 osd.3 since back 2024-04-15T21:54:34.265916+0000 front 2024-04-15T21:54:34.265850+0000 (oldest deadline 2024-04-15T21:54:59.565735+0000) 2024-04-15T21:55:05.523 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:05 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7[37796]: 2024-04-15T21:55:05.269+0000 7f333f032700 -1 osd.7 96 heartbeat_check: no reply from 172.21.15.33:6804 osd.4 since back 2024-04-15T21:54:39.566252+0000 front 2024-04-15T21:54:39.566171+0000 (oldest deadline 2024-04-15T21:55:04.866210+0000) 2024-04-15T21:55:06.522 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:06 smithi033 podman[47273]: 2024-04-15 21:55:06.323636574 +0000 UTC m=+5.540126759 container died 2d7772ed29bc865fc848f1e5d41a61c6a267a0228b124edb8ffa3f297fc35c71 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, io.openshift.expose-services=, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, GIT_CLEAN=True, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, ceph=True, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754) 2024-04-15T21:55:06.871 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:06 smithi033 podman[47273]: 2024-04-15 21:55:06.760629695 +0000 UTC m=+5.977119886 container cleanup 2d7772ed29bc865fc848f1e5d41a61c6a267a0228b124edb8ffa3f297fc35c71 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7, io.k8s.display-name=CentOS Stream 8, vcs-type=git, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.openshift.expose-services=, ceph=True, name=centos-stream, GIT_CLEAN=True, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , distribution-scope=public, release=754, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-15T21:55:06.871 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:06 smithi033 bash[47273]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7 2024-04-15T21:55:07.189 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:06 smithi033 podman[47301]: 2024-04-15 21:55:06.95377638 +0000 UTC m=+0.622873317 container remove 2d7772ed29bc865fc848f1e5d41a61c6a267a0228b124edb8ffa3f297fc35c71 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, name=centos-stream, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=) 2024-04-15T21:55:07.190 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:06 smithi033 bash[47313]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd.7" found: no such container 2024-04-15T21:55:07.474 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:07 smithi033 podman[47380]: 2024-04-15 21:55:07.092268217 +0000 UTC m=+0.017975067 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-15T21:55:07.474 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:07 smithi033 podman[47380]: 2024-04-15 21:55:07.196092965 +0000 UTC m=+0.121799841 container create 1335a2ca94f6e5f737cba4113ad89a5c3f89b49ab02ab22f880c7d4bab1e7301 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7-deactivate, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, architecture=x86_64, io.openshift.expose-services=, vendor=Red Hat, Inc., distribution-scope=public, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, version=8, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , vcs-type=git, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-15T21:55:07.475 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:07 smithi033 podman[47380]: 2024-04-15 21:55:07.398259506 +0000 UTC m=+0.323966340 container init 1335a2ca94f6e5f737cba4113ad89a5c3f89b49ab02ab22f880c7d4bab1e7301 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7-deactivate, RELEASE=HEAD, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, ceph=True, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, version=8, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-15T21:55:07.475 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:07 smithi033 podman[47380]: 2024-04-15 21:55:07.401517734 +0000 UTC m=+0.327224564 container start 1335a2ca94f6e5f737cba4113ad89a5c3f89b49ab02ab22f880c7d4bab1e7301 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, RELEASE=HEAD, io.openshift.expose-services=, distribution-scope=public, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, version=8, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, vcs-type=git, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , ceph=True) 2024-04-15T21:55:07.475 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:07 smithi033 podman[47380]: 2024-04-15 21:55:07.473198151 +0000 UTC m=+0.398904994 container attach 1335a2ca94f6e5f737cba4113ad89a5c3f89b49ab02ab22f880c7d4bab1e7301 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7-deactivate, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, io.openshift.expose-services=, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, vendor=Red Hat, Inc., distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True) 2024-04-15T21:55:07.772 INFO:journalctl@ceph.osd.7.smithi033.stdout:Apr 15 21:55:07 smithi033 podman[47380]: 2024-04-15 21:55:07.690809006 +0000 UTC m=+0.616515831 container died 1335a2ca94f6e5f737cba4113ad89a5c3f89b49ab02ab22f880c7d4bab1e7301 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-osd-7-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.openshift.expose-services=, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.buildah.version=1.19.8) 2024-04-15T21:55:08.339 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@osd.7.service' 2024-04-15T21:55:08.382 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:55:08.383 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-15T21:55:08.383 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-04-15T21:55:08.383 DEBUG:teuthology.orchestra.run.smithi033:> sudo systemctl stop ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@prometheus.a 2024-04-15T21:55:08.589 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 systemd[1]: Stopping Ceph prometheus.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:55:08.589 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 bash[47458]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus.a" found: no such container 2024-04-15T21:55:08.863 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:55:08.588Z caller=main.go:775 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-04-15T21:55:08.863 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:55:08.588Z caller=main.go:798 level=info msg="Stopping scrape discovery manager..." 2024-04-15T21:55:08.863 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:55:08.588Z caller=main.go:812 level=info msg="Stopping notify discovery manager..." 2024-04-15T21:55:08.863 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:55:08.588Z caller=main.go:834 level=info msg="Stopping scrape manager..." 2024-04-15T21:55:08.863 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:55:08.588Z caller=main.go:808 level=info msg="Notify discovery manager stopped" 2024-04-15T21:55:08.863 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:55:08.588Z caller=main.go:794 level=info msg="Scrape discovery manager stopped" 2024-04-15T21:55:08.864 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:55:08.588Z caller=manager.go:945 level=info component="rule manager" msg="Stopping rule manager..." 2024-04-15T21:55:08.864 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:55:08.588Z caller=main.go:828 level=info msg="Scrape manager stopped" 2024-04-15T21:55:08.864 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:55:08.588Z caller=manager.go:955 level=info component="rule manager" msg="Rule manager stopped" 2024-04-15T21:55:08.864 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:55:08.588Z caller=notifier.go:600 level=info component=notifier msg="Stopping notification manager..." 2024-04-15T21:55:08.864 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:55:08.588Z caller=main.go:1054 level=info msg="Notifier manager stopped" 2024-04-15T21:55:08.864 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a[42577]: ts=2024-04-15T21:55:08.588Z caller=main.go:1066 level=info msg="See you next time!" 2024-04-15T21:55:08.864 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 podman[47470]: 2024-04-15 21:55:08.598550228 +0000 UTC m=+0.101628584 container died 2102c8d49f6e2bacafafec3d992c6d112b47c294031e74a162fc9aa0c298740e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T21:55:09.204 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 podman[47470]: 2024-04-15 21:55:08.961195568 +0000 UTC m=+0.464273929 container cleanup 2102c8d49f6e2bacafafec3d992c6d112b47c294031e74a162fc9aa0c298740e (image=quay.io/prometheus/prometheus:v2.33.4, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-04-15T21:55:09.204 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 bash[47470]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus-a 2024-04-15T21:55:09.205 INFO:journalctl@ceph.prometheus.a.smithi033.stdout:Apr 15 21:55:08 smithi033 bash[47497]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-prometheus.a" found: no such container 2024-04-15T21:55:09.209 DEBUG:teuthology.orchestra.run.smithi033:> sudo pkill -f 'journalctl -f -n 0 -u ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@prometheus.a.service' 2024-04-15T21:55:09.247 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-15T21:55:09.247 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-04-15T21:55:09.247 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-15T21:55:10.956 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:55:10 smithi007 systemd[1]: Stopping Ceph node-exporter.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:55:10.956 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:55:10 smithi007 bash[58567]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter.a" found: no such container 2024-04-15T21:55:11.292 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:55:10 smithi007 podman[58579]: 2024-04-15 21:55:10.954592691 +0000 UTC m=+0.096692579 container died 49d5ffb0b3d0bf83929da50acc402be5da2e113b2b8e528349e7237ca5e48799 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-15T21:55:11.662 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:55:11 smithi007 podman[58579]: 2024-04-15 21:55:11.485653642 +0000 UTC m=+0.627753519 container cleanup 49d5ffb0b3d0bf83929da50acc402be5da2e113b2b8e528349e7237ca5e48799 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-15T21:55:11.662 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:55:11 smithi007 bash[58579]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a 2024-04-15T21:55:12.007 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:55:11 smithi007 systemd[1]: Stopping Ceph alertmanager.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:55:12.008 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:55:11 smithi007 bash[58659]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager.a" found: no such container 2024-04-15T21:55:12.008 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:55:11 smithi007 podman[58593]: 2024-04-15 21:55:11.66168835 +0000 UTC m=+0.709678903 container remove 49d5ffb0b3d0bf83929da50acc402be5da2e113b2b8e528349e7237ca5e48799 (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-04-15T21:55:12.008 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:55:11 smithi007 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-04-15T21:55:12.008 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:55:11 smithi007 bash[58606]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter.a" found: no such container 2024-04-15T21:55:12.008 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:55:11 smithi007 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-04-15T21:55:12.008 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:55:11 smithi007 systemd[1]: Stopped Ceph node-exporter.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:55:12.008 INFO:journalctl@ceph.node-exporter.a.smithi007.stdout:Apr 15 21:55:11 smithi007 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@node-exporter.a.service: Consumed 3.032s CPU time. 2024-04-15T21:55:12.274 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:55:12 smithi007 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a[49611]: level=info ts=2024-04-15T21:55:12.006Z caller=main.go:557 msg="Received SIGTERM, exiting gracefully..." 2024-04-15T21:55:12.274 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:55:12 smithi007 podman[58670]: 2024-04-15 21:55:12.037344762 +0000 UTC m=+0.121063710 container died 9aff9a2b17564d335a36252f682389279da2e17d10d40bddd5510f7097cadd5a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T21:55:12.541 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:55:12 smithi007 podman[58670]: 2024-04-15 21:55:12.373136217 +0000 UTC m=+0.456855163 container cleanup 9aff9a2b17564d335a36252f682389279da2e17d10d40bddd5510f7097cadd5a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T21:55:12.542 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:55:12 smithi007 bash[58670]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a 2024-04-15T21:55:12.542 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:55:12 smithi007 bash[58699]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager.a" found: no such container 2024-04-15T21:55:12.902 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:55:12 smithi007 podman[58685]: 2024-04-15 21:55:12.640658384 +0000 UTC m=+0.600234496 container remove 9aff9a2b17564d335a36252f682389279da2e17d10d40bddd5510f7097cadd5a (image=quay.io/prometheus/alertmanager:v0.23.0, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-04-15T21:55:12.902 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:55:12 smithi007 podman[58685]: 2024-04-15 21:55:12.708321878 +0000 UTC m=+0.667897990 volume remove 37d00418590f7641e540a5b0ef496ebd73615bfdca191e684fa6a7f11b95d47a 2024-04-15T21:55:12.902 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:55:12 smithi007 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-04-15T21:55:12.902 INFO:journalctl@ceph.alertmanager.a.smithi007.stdout:Apr 15 21:55:12 smithi007 systemd[1]: Stopped Ceph alertmanager.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:55:46.592 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 --force --keep-logs 2024-04-15T21:55:48.523 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:55:48 smithi033 systemd[1]: Stopping Ceph node-exporter.b for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:55:48.523 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:55:48 smithi033 bash[47826]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter.b" found: no such container 2024-04-15T21:55:48.523 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:55:48 smithi033 podman[47839]: 2024-04-15 21:55:48.204597843 +0000 UTC m=+0.108486900 container died 80c30792c19be9b42cc513d39ab44a136f4cc420d34191ab7907e592beaff30e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-15T21:55:48.987 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:55:48 smithi033 podman[47839]: 2024-04-15 21:55:48.73567674 +0000 UTC m=+0.639565838 container cleanup 80c30792c19be9b42cc513d39ab44a136f4cc420d34191ab7907e592beaff30e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-15T21:55:48.987 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:55:48 smithi033 bash[47839]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b 2024-04-15T21:55:49.267 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:55:49 smithi033 systemd[1]: Stopping Ceph grafana.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968... 2024-04-15T21:55:49.268 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:55:49 smithi033 bash[47922]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana.a" found: no such container 2024-04-15T21:55:49.268 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:55:48 smithi033 podman[47854]: 2024-04-15 21:55:48.986683751 +0000 UTC m=+0.786054629 container remove 80c30792c19be9b42cc513d39ab44a136f4cc420d34191ab7907e592beaff30e (image=quay.io/prometheus/node-exporter:v1.3.1, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-04-15T21:55:49.268 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:55:48 smithi033 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-04-15T21:55:49.268 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:55:48 smithi033 bash[47867]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-node-exporter.b" found: no such container 2024-04-15T21:55:49.268 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:55:49 smithi033 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-04-15T21:55:49.268 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:55:49 smithi033 systemd[1]: Stopped Ceph node-exporter.b for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:55:49.269 INFO:journalctl@ceph.node-exporter.b.smithi033.stdout:Apr 15 21:55:49 smithi033 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@node-exporter.b.service: Consumed 2.997s CPU time. 2024-04-15T21:55:49.522 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:55:49 smithi033 ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a[41771]: t=2024-04-15T21:55:49+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated" 2024-04-15T21:55:49.522 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:55:49 smithi033 podman[47932]: 2024-04-15 21:55:49.375617223 +0000 UTC m=+0.135982399 container died ba919d83187eb1802cd94c4660cf4800e8ae16465205a6dc46045da3a7720f80 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a, description=Ceph Grafana Container, vcs-type=git, vendor=Red Hat, Inc., architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, build-date=2022-03-28T10:36:18.413762, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.24.2, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, version=8.5, io.openshift.tags=base rhel8, release=236.1648460182) 2024-04-15T21:55:49.899 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:55:49 smithi033 podman[47932]: 2024-04-15 21:55:49.697665387 +0000 UTC m=+0.458030552 container cleanup ba919d83187eb1802cd94c4660cf4800e8ae16465205a6dc46045da3a7720f80 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a, vendor=Red Hat, Inc., name=ubi8, build-date=2022-03-28T10:36:18.413762, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, io.buildah.version=1.24.2, io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, com.redhat.component=ubi8-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , release=236.1648460182, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, io.openshift.expose-services=, io.openshift.tags=base rhel8, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container) 2024-04-15T21:55:49.899 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:55:49 smithi033 bash[47932]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a 2024-04-15T21:55:49.899 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:55:49 smithi033 bash[47968]: Error: no container with name or ID "ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana.a" found: no such container 2024-04-15T21:55:50.211 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:55:49 smithi033 podman[47949]: 2024-04-15 21:55:49.898619487 +0000 UTC m=+0.522409243 container remove ba919d83187eb1802cd94c4660cf4800e8ae16465205a6dc46045da3a7720f80 (image=quay.io/ceph/ceph-grafana:8.3.5, name=ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968-grafana-a, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., architecture=x86_64, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-236.1648460182, com.redhat.component=ubi8-container, io.buildah.version=1.24.2, distribution-scope=public, build-date=2022-03-28T10:36:18.413762, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, name=ubi8, release=236.1648460182) 2024-04-15T21:55:50.212 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:55:49 smithi033 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@grafana.a.service: Deactivated successfully. 2024-04-15T21:55:50.212 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:55:49 smithi033 systemd[1]: Stopped Ceph grafana.a for 18ed60a0-fb71-11ee-bc8f-c7b262605968. 2024-04-15T21:55:50.212 INFO:journalctl@ceph.grafana.a.smithi033.stdout:Apr 15 21:55:49 smithi033 systemd[1]: ceph-18ed60a0-fb71-11ee-bc8f-c7b262605968@grafana.a.service: Consumed 2.729s CPU time. 2024-04-15T21:56:12.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T21:56:12.400 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-15T21:56:12.431 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-15T21:56:12.433 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657477/remote/smithi007/crash 2024-04-15T21:56:12.434 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/crash -- . 2024-04-15T21:56:12.486 INFO:teuthology.orchestra.run.smithi007.stderr:tar: /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-15T21:56:12.489 INFO:teuthology.orchestra.run.smithi007.stderr:tar: Error is not recoverable: exiting now 2024-04-15T21:56:12.491 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657477/remote/smithi033/crash 2024-04-15T21:56:12.492 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/crash -- . 2024-04-15T21:56:12.534 INFO:teuthology.orchestra.run.smithi033.stderr:tar: /var/lib/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/crash: Cannot open: No such file or directory 2024-04-15T21:56:12.534 INFO:teuthology.orchestra.run.smithi033.stderr:tar: Error is not recoverable: exiting now 2024-04-15T21:56:12.535 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-15T21:56:12.535 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-15T21:56:12.588 INFO:teuthology.orchestra.run.smithi007.stdout:2024-04-15T21:44:57.627362+0000 mon.a (mon.0) 168 : cluster [WRN] Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-04-15T21:56:12.589 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-15T21:56:12.589 INFO:tasks.cephadm:Compressing logs... 2024-04-15T21:56:12.589 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T21:56:12.591 DEBUG:teuthology.orchestra.run.smithi033:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-15T21:56:12.625 INFO:teuthology.orchestra.run.smithi033.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose --: No such file or directory /var/log/ceph/cephadm.log 2024-04-15T21:56:12.625 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-04-15T21:56:12.625 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-client.rgw.smpl.smithi033.tenllr.log 2024-04-15T21:56:12.625 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.6.log 2024-04-15T21:56:12.626 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-client.rgw.smpl.smithi033.tenllr.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-15T21:56:12.626 INFO:teuthology.orchestra.run.smithi033.stderr: 62.5% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-client.rgw.smpl.smithi033.tenllr.log.gz 2024-04-15T21:56:12.626 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.4.log 2024-04-15T21:56:12.626 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-client.rgw.foo.smithi033.glebqt.log 2024-04-15T21:56:12.626 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.7.log 2024-04-15T21:56:12.626 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.4.log: /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-client.rgw.foo.smithi033.glebqt.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.5.log 2024-04-15T21:56:12.626 INFO:teuthology.orchestra.run.smithi033.stderr: 62.5% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-client.rgw.foo.smithi033.glebqt.log.gz 2024-04-15T21:56:12.626 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mgr.x.log 2024-04-15T21:56:12.627 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-15T21:56:12.627 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.log 2024-04-15T21:56:12.627 INFO:teuthology.orchestra.run.smithi033.stderr: 90.1% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-15T21:56:12.627 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-15T21:56:12.627 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.log: 79.9% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-15T21:56:12.627 INFO:teuthology.orchestra.run.smithi033.stderr:gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mon.b.log 2024-04-15T21:56:12.627 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-volume.log: 87.3% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-15T21:56:12.627 INFO:teuthology.orchestra.run.smithi033.stderr: 90.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-15T21:56:12.629 INFO:teuthology.orchestra.run.smithi033.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mon.b.log: 91.1% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mgr.x.log.gz 2024-04-15T21:56:12.636 INFO:teuthology.orchestra.run.smithi007.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-15T21:56:12.636 INFO:teuthology.orchestra.run.smithi007.stderr:: No such file or directory 2024-04-15T21:56:12.636 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mon.a.log 2024-04-15T21:56:12.636 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.audit.log 2024-04-15T21:56:12.636 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.1.log 2024-04-15T21:56:12.637 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mon.a.log: /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.3.log 2024-04-15T21:56:12.637 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-client.rgw.foo.smithi007.qlizaq.log 2024-04-15T21:56:12.637 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mgr.y.log 2024-04-15T21:56:12.637 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-client.rgw.foo.smithi007.qlizaq.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-client.rgw.smpl.smithi007.uhnuit.log 2024-04-15T21:56:12.637 INFO:teuthology.orchestra.run.smithi007.stderr: 62.5% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-client.rgw.foo.smithi007.qlizaq.log.gz 2024-04-15T21:56:12.637 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/tcmu-runner.log 2024-04-15T21:56:12.637 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-client.rgw.smpl.smithi007.uhnuit.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.0.log 2024-04-15T21:56:12.637 INFO:teuthology.orchestra.run.smithi007.stderr: 62.4% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-client.rgw.smpl.smithi007.uhnuit.log.gz 2024-04-15T21:56:12.638 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/tcmu-runner.log: 66.6% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/tcmu-runner.log.gz 2024-04-15T21:56:12.638 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.cephadm.log 2024-04-15T21:56:12.638 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.0.log: 93.8% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.audit.log.gz 2024-04-15T21:56:12.638 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.log 2024-04-15T21:56:12.638 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.2.log 2024-04-15T21:56:12.638 INFO:teuthology.orchestra.run.smithi007.stderr: 88.4% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.cephadm.log.gz 2024-04-15T21:56:12.638 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-volume.log 2024-04-15T21:56:12.638 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.2.log: 92.7% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph.log.gz 2024-04-15T21:56:12.638 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mon.c.log 2024-04-15T21:56:12.639 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-volume.log: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-15T21:56:12.639 INFO:teuthology.orchestra.run.smithi007.stderr: 91.1% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-15T21:56:12.639 INFO:teuthology.orchestra.run.smithi033.stderr: 90.9% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-volume.log.gz 2024-04-15T21:56:12.738 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mon.c.log: 88.9% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mgr.y.log.gz 2024-04-15T21:56:12.760 INFO:teuthology.orchestra.run.smithi033.stderr: 92.2% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mon.b.log.gz 2024-04-15T21:56:12.882 INFO:teuthology.orchestra.run.smithi007.stderr: 92.0% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mon.c.log.gz 2024-04-15T21:56:12.923 INFO:teuthology.orchestra.run.smithi033.stderr: 93.2% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.6.log.gz 2024-04-15T21:56:12.967 INFO:teuthology.orchestra.run.smithi007.stderr: 93.4% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.2.log.gz 2024-04-15T21:56:13.003 INFO:teuthology.orchestra.run.smithi033.stderr: 93.5% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.5.log.gz 2024-04-15T21:56:13.059 INFO:teuthology.orchestra.run.smithi007.stderr: 93.5% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.0.log.gz 2024-04-15T21:56:13.092 INFO:teuthology.orchestra.run.smithi033.stderr: 93.7% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.7.log.gz 2024-04-15T21:56:13.100 INFO:teuthology.orchestra.run.smithi007.stderr: 90.9% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-mon.a.log.gz 2024-04-15T21:56:13.108 INFO:teuthology.orchestra.run.smithi007.stderr: 93.6% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.1.log.gz 2024-04-15T21:56:13.116 INFO:teuthology.orchestra.run.smithi033.stderr: 93.6% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.4.log.gz 2024-04-15T21:56:13.118 INFO:teuthology.orchestra.run.smithi033.stderr: 2024-04-15T21:56:13.118 INFO:teuthology.orchestra.run.smithi033.stderr:real 0m0.512s 2024-04-15T21:56:13.118 INFO:teuthology.orchestra.run.smithi033.stderr:user 0m1.765s 2024-04-15T21:56:13.118 INFO:teuthology.orchestra.run.smithi033.stderr:sys 0m0.080s 2024-04-15T21:56:13.132 INFO:teuthology.orchestra.run.smithi007.stderr: 93.6% -- replaced with /var/log/ceph/18ed60a0-fb71-11ee-bc8f-c7b262605968/ceph-osd.3.log.gz 2024-04-15T21:56:13.134 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-04-15T21:56:13.134 INFO:teuthology.orchestra.run.smithi007.stderr:real 0m0.529s 2024-04-15T21:56:13.134 INFO:teuthology.orchestra.run.smithi007.stderr:user 0m2.574s 2024-04-15T21:56:13.134 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m0.107s 2024-04-15T21:56:13.134 INFO:tasks.cephadm:Archiving logs... 2024-04-15T21:56:13.135 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657477/remote/smithi007/log 2024-04-15T21:56:13.136 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T21:56:13.538 DEBUG:teuthology.misc:Transferring archived files from smithi033:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657477/remote/smithi033/log 2024-04-15T21:56:13.539 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-15T21:56:13.843 INFO:tasks.cephadm:Removing cluster... 2024-04-15T21:56:13.844 DEBUG:teuthology.orchestra.run.smithi007:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 --force 2024-04-15T21:56:14.129 DEBUG:teuthology.orchestra.run.smithi033:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 --force 2024-04-15T21:56:14.422 INFO:tasks.cephadm:Removing cephadm ... 2024-04-15T21:56:14.423 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-15T21:56:14.440 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-15T21:56:14.457 INFO:tasks.cephadm:Teardown complete 2024-04-15T21:56:14.457 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-15T21:56:14.466 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-15T21:56:14.467 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T21:56:14.483 DEBUG:teuthology.orchestra.run.smithi033:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-15T21:56:14.495 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-04-15T21:56:14.499 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T21:56:14.499 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-04-15T21:56:14.499 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv01.front.sepia.ceph.com 3 6 377 25 +888us[ +901us] +/- 44ms 2024-04-15T21:56:14.500 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 24 -390us[ -390us] +/- 58ms 2024-04-15T21:56:14.500 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 28 -950us[ -937us] +/- 70ms 2024-04-15T21:56:14.500 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T21:56:14.510 INFO:teuthology.orchestra.run.smithi033.stderr:bash: line 1: ntpq: command not found 2024-04-15T21:56:14.514 INFO:teuthology.orchestra.run.smithi033.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-15T21:56:14.514 INFO:teuthology.orchestra.run.smithi033.stdout:=============================================================================== 2024-04-15T21:56:14.514 INFO:teuthology.orchestra.run.smithi033.stdout:^* hv01.front.sepia.ceph.com 3 6 377 25 +895us[ +901us] +/- 44ms 2024-04-15T21:56:14.514 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 25 -396us[ -389us] +/- 58ms 2024-04-15T21:56:14.514 INFO:teuthology.orchestra.run.smithi033.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 -933us[ -933us] +/- 70ms 2024-04-15T21:56:14.515 INFO:teuthology.orchestra.run.smithi033.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-15T21:56:14.516 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-15T21:56:14.525 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-15T21:56:14.526 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-15T21:56:14.537 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T21:56:14.620 DEBUG:teuthology.orchestra.run.smithi033:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-15T21:56:14.693 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-15T21:56:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T21:56:14.725 DEBUG:teuthology.orchestra.run.smithi033:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-04-15T21:56:14.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T21:56:14.756 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-15T21:56:14.765 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-15T21:56:14.824 INFO:teuthology.task.internal:Duration was 1081.869604 seconds 2024-04-15T21:56:14.825 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-15T21:56:14.834 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-15T21:56:14.834 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T21:56:14.837 DEBUG:teuthology.orchestra.run.smithi033:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-15T21:56:14.881 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T21:56:14.882 INFO:teuthology.orchestra.run.smithi033.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-15T21:56:15.327 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-15T21:56:15.327 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-04-15T21:56:15.327 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T21:56:15.348 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi033.front.sepia.ceph.com 2024-04-15T21:56:15.348 DEBUG:teuthology.orchestra.run.smithi033:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-15T21:56:15.375 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-15T21:56:15.375 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T21:56:15.391 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-15T21:56:15.445 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-15T21:56:15.445 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T21:56:15.461 DEBUG:teuthology.orchestra.run.smithi033:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-15T21:56:15.756 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-15T21:56:15.768 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-15T21:56:15.769 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T21:56:15.798 DEBUG:teuthology.orchestra.run.smithi033:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-15T21:56:15.827 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-15T21:56:15.895 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T21:56:15.897 DEBUG:teuthology.orchestra.run.smithi033:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-15T21:56:15.922 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-04-15T21:56:15.924 INFO:teuthology.orchestra.run.smithi033.stdout:kernel.core_pattern = core 2024-04-15T21:56:15.960 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T21:56:16.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T21:56:16.001 DEBUG:teuthology.orchestra.run.smithi033:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-15T21:56:16.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-15T21:56:16.016 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-15T21:56:16.028 INFO:teuthology.task.internal:Transferring archived files... 2024-04-15T21:56:16.029 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657477/remote/smithi007 2024-04-15T21:56:16.029 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T21:56:16.086 DEBUG:teuthology.misc:Transferring archived files from smithi033:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657477/remote/smithi033 2024-04-15T21:56:16.087 DEBUG:teuthology.orchestra.run.smithi033:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-15T21:56:16.127 INFO:teuthology.task.internal:Removing archive directory... 2024-04-15T21:56:16.128 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T21:56:16.130 DEBUG:teuthology.orchestra.run.smithi033:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-15T21:56:16.176 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-15T21:56:16.187 INFO:teuthology.task.internal:Not uploading archives. 2024-04-15T21:56:16.187 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-15T21:56:16.197 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-15T21:56:16.198 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T21:56:16.200 DEBUG:teuthology.orchestra.run.smithi033:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-15T21:56:16.212 INFO:teuthology.orchestra.run.smithi007.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 21:56 /home/ubuntu/cephtest 2024-04-15T21:56:16.230 INFO:teuthology.orchestra.run.smithi033.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 15 21:56 /home/ubuntu/cephtest 2024-04-15T21:56:16.231 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-15T21:56:16.242 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-15T21:56:16.262 INFO:teuthology.nuke:Checking targets against current locks 2024-04-15T21:56:16.292 DEBUG:teuthology.nuke:shortname: smithi007 2024-04-15T21:56:16.293 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T21:56:16.317 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657477', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 21:30:28.093615', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPwmQiH618WrhD7w/h2f5W0qF3ls5QPMhWNPeYdcZr4q4iT15pBHmc0403j29s+ld3WDFLSkKpMLJpnXpD/9vb4='} 2024-04-15T21:56:16.321 DEBUG:teuthology.nuke:shortname: smithi033 2024-04-15T21:56:16.321 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-15T21:56:16.340 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi033.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-15_21:08:03-orch-squid-distro-default-smithi/7657477', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-15 21:30:28.092212', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBECNfbdraZfwlGxP5ilb6Y0oCb66kvkAJqLd2/ZRoNcNbQXVxHTIbnj5O3iwaBvFljqZCLlwJCN5/sB23No0wYA='} 2024-04-15T21:56:16.375 INFO:teuthology.orchestra.console.smithi007:Power off 2024-04-15T21:56:16.376 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T21:56:16.392 INFO:teuthology.orchestra.console.smithi033:Power off 2024-04-15T21:56:16.393 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-15T21:56:16.406 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2024-04-15T21:56:16.407 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T21:56:16.420 DEBUG:teuthology.orchestra.console.smithi033:power off output: Chassis Power Control: Down/Off 2024-04-15T21:56:16.420 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T21:56:16.433 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-04-15T21:56:16.446 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-04-15T21:56:20.435 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T21:56:20.449 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T21:56:20.551 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-04-15T21:56:20.563 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-04-15T21:56:24.558 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T21:56:24.568 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T21:56:24.670 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-04-15T21:56:24.680 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is on 2024-04-15T21:56:24.772 INFO:teuthology.orchestra.console.smithi007:Power off completed 2024-04-15T21:56:28.682 DEBUG:teuthology.orchestra.console.smithi033:pexpect command: ipmitool -H smithi033.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-15T21:56:28.792 DEBUG:teuthology.orchestra.console.smithi033:check power output: Chassis Power is off 2024-04-15T21:56:28.894 INFO:teuthology.orchestra.console.smithi033:Power off completed 2024-04-15T21:56:28.999 INFO:teuthology.run:Summary data: description: orch/cephadm/upgrade/{1-start-distro/1-start-centos_9.stream 2-repo_digest/defaut 3-upgrade/simple 4-wait 5-upgrade-ls agent/off mon_election/connectivity} duration: 1081.8696036338806 failure_reason: 'Command failed on smithi007 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 18ed60a0-fb71-11ee-bc8f-c7b262605968 -e sha1=a9a752df26c63acad72e1b3569fd79a515ca0765 -- bash -c \''ceph versions | jq -e \''"\''"\''.overall | keys\''"\''"\'' | grep $sha1\''''' owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=2f4d5abd2f49463591127c9a995a0444 status: fail success: false 2024-04-15T21:56:28.999 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-15T21:56:29.086 INFO:teuthology.run:FAIL